


‘ 


nd LIPA 






Institutional Archive of the Naval Postgraduate School 


Calhoun: The NPS Institutional Archive 
DSpace Repository 


Theses and Dissertations 1. Thesis and Dissertation Collection, all items 


1987-12 


Fault diagnosis in distributed computer networks 


Dincer, lbrahim. 


http://ndl.handle.net/10945/22807 
Copyright is reserved by the copyright owner 


Downloaded from NPS Archive: Calhoun 


| Calhoun is the Naval Postgraduate School's public access digital repository for 
(8 D U DLEY research materials and institutional publications created by the NPS community. 
«ist | | Calhoun is named for Professor of Mathematics Guy K. Calhoun, NPS's first 


1 i KNOX appointed — and published -- scholarly author. 

ih : Dudley Knox Libra Naval Postgraduate Schoo 
LIBRARY dley b ry i | g d hool 

http://www.nps.edu/library 






411 Dyer Road / 1 University Circle 
Monterey, California USA 93943 


. 


ee ed 


















-? pie reyes Re bay: v was 
a Pwr, 
4 3 Tr 3 4.# Ve eh, rig tri Ne Cate ond aa , 
AJ ‘ Me * 7 Pree De OS matic ae i vt 
| oe J I ) A Pn 4% ; * ry Ai ‘ ee I fy = i a sie vb mt bya ra 
' ‘ 4 4 + 
; ' LE WEL. ea ata ee is SNR AY rc Tenis ; Se erie he 
- / 4 Ls . 4 Vo J ~ fe ory r a M iY s | MPF e beterte a bs et "319 
ee , . SEARLS rama exe ee Bane near ace 
fe » Yee of ws ere Lee pectin ‘wee pets by CAL We ie WAP er vie 
s * *y RON eon ring ge a TU en 
4 - om i a A 4 ie oats ‘ 4 Ur EF 
D E . yl Phy ie yy A 
r ' a we ick Pie br a Pave Oh nn a * a mit Wd rae tr geil 
Para | y i a: Di tara ne Tag % NE ee eh eons A 
‘ 5 . y Ve Ly ipdy ae re rh ToC Suet 
r r ar ry Py a AL . ri en + vee n% A+ ee “att 7 NO Reet 
f ‘4 ae “ ‘, a were. “& va Retest Epes fas) Baan Pr Rone ror alee 
| a : yell SA AL SSS Crh NOY aca 
FS " 14 ek Piet ee Gate " Hoey: La Cy ve ye Lh 
F $ F ; Y ran kaha 4g wy Re he Ara ca Se mee 
5 , ; _ D Pres rt Cree Wa Pet ie hgh ryzh Nera ees 
E J = F he whe oh ‘ Lia : 4 Yh py my ara ry Ar sat 
i f » , Oe ie On ee ri aL ht 3 
; a a ate a he fe Oc iM $ i ie WA i sere iy eye 
. — dee tL HSeparkee Bet on > ey hs 
A ® ' ae Ae L} mn te lbodes Sti bie Wea yer TET ea a oem rr ea ee 
; : x) Pare here’ ° aA iby mata eter Ores un ree ns eee 
j ss ry 4 fey $4 4 ater ta eh. ff trig BY he 
D e s F Hl ‘ Alea yy $IPatent BaP da) Veber Wj 4 paler Soe cen 
ae ee earn ME een LT WE eo et Pele treba a ert neta * 
ea ‘ ' rrF: Care, Byrn ‘yy eS bey Tena EA eet; 
; 5 ie vr A of hod , 4 deg paar 
ST ERS ee SR Rs | mas on a Se ey eae CuRVaR Aor e 
; at re He TE} non ae te aoe ah bY Sree ot oats vpn) ppt Pettey 
: hi F . . a. a Cert chek yp MR retards hor Pie a 
Ke’ : aon a ‘i ra nH y < ely? > rac By rege vy ye 4 es rh rag So Se Ae 
a * B14 4 yale . rv aay o ee Ie Cea " w 
4 - A wy » a ‘ - Peete mS a vy i a 
ay Par way A ve brs y ‘ ast aon rn 4h mp tignts ro oa sXe cays: 
" i Wa AY “ee ; ME al a Mook if ‘ BS 7 rie A tg %;) nN Re sry tn Aaa e a 
- , - , , t f a, i oa, s P be or ee ; Oy ey Soa) 
. a , toy oe pier oe ci mt" “ ane Sah pal 0 OS 8 tere ice Pe See tte ’ 
i 1 fie J apap a) ee on re ate » Wo rin five Wise eth CTEM re 
a ; ar ree py mol ae ricer Doras ea to eg —— 
, , zy . ; i i 9 hs y Ne " Th ee 
. 4 | ey | iW . ; 


sy 
7 ‘ L* 7 he | a | va on 7 
" L?) 4 ¢ 
' vy vi } 
‘| P Ac o 5 . 


bea! 
Ml vr Lt 
. aan pete eS See Lt Srntentea 
NS SU Wie Amik, ee Rsereaee at. ae Pret ry eer wigs aaal 4 eae — 
SE Ye Se PA re pete a PN pare hers ere iebayrwegdlel alent eee Crt pcr. 
1 Jae a = Feyde “Ni 4 7] pT A, A ore a 7A 
. Cie) ZI n . » ‘ 
: at PK ' LD 


Le ee re Wa i) DG re Pah AED. Ne AGMA O he ig: abba to ee ee 
4 r sr vy ra ear Ao: art re ae Ra Cate! i NG ane oc] ; 
mH } a Apartado A a opie. ths iat agae 
a “i orm, oe OT , wr ei 
i 4 a es oe % Bc : i ¥ a * 4 aaeue ! i + 
' A i ti 






















































































































Ae cae 
Ot eee mY ca tee pk, 
A ee ry ae Petri ern ttn ‘i a erie Ra ae ti pac er r 
: at Te CR ra Se Sie i bs Bees rh area vy as borates = tins es 
ar | : 4 a A o : - ra LT si reer Per o®'hAeat eis al jae ap 
5 o Mies yc 4 ‘ 2 ath # i 5 arn, Ss or USe ae ree a Mee hier Ls ear tr a ant 
Las eee o i a hy - A ? ie an ie s ~ or ar war] re Aa ort ae? AW A iy 8 $ ee sey a: ANTES BCs Stee ied! = 
n 5 wae Pa om: 4 fot A ml) ne sa oy ALY 4 Fe ee A hN aaa 
7 ‘ oI i é i, . 7 A 
, ae y : wo OR CO Feet ee § a oe nt My oe rie ors an ni A eri rics 
re Mee Lege. 1 OS y Oe uaa OY joe t pe yaa reas eit re Mohr oy ae eer fs eat 
we 4 “ee ae Uh ils re Tyga h, ae Lary VAY ee a yee Fey el iareeh sw be Wye ce ing iy Perv, ty epg 
' 5 + & peat rte i ‘ iy 4 - ‘i f ~~ a r © 7 Pree) Reet ACLS, i bebe te Oh ard ry 
iO P rT s A . 7 } of 
a a yee ernie) Sea alae air tis eon ee atb oman 
. . , A ‘ iM “a \. “, 
; a | ig : : 7} . qe : n = ‘| . i ei sac ni wre 5 tag é wed sy i ir Piha were rere arti baat 
A eth A é AR : a 4 a . f hy | i ; r =~ ae a rae a) rr me rs ba 7 rer obese arr be ." a 2C ony wi > a an Rif By 
: i ee ‘al oS OE 1 om! ee ay Sat ac PRA Mor bet Lin lh a ry tegs oan tees eI Cray ad en Cs abt 
a aC Se ee f la ; 1 eae) eres etmek ieee bh est DN Poni eek ee. ayes Sk 
p y t va om, f > Fi " me Ae ee ee Seerela hw ee testo Vr oe, vs 
A ae qiuook iT he ti , bs be BOL A Rekd 8 Fae a ee) ho bk eon by 
o i ‘ '» i 5 ‘ é wa by ee fi ee ee 4a % beans PLL a wei Masts: pe ae bt C0 
4 ; tee t LY ro ' ie ‘i De ot a ae TIU TY (Va V Y vp te Ph oe inree ms Y 
5 fe ona acy We)  * =; ; ee ‘ior ee ori +k rn ot pay us ie b 
aoe a | tua ee) Boat Pee Sa Sa nA ie ty Pas bli taet te weg um eatin e et "i *. ath Raa oe 
1 . . ‘ ‘ 7 q r F : Py 4 re rr 
a , ; ve i F - A eee | ae Le ee a ia ix a Be ae mone aa toe Pew pert rere Si Neier at Buse 
$ "| A me ty ' ds en | ? A : 7 ds EMT NFAY Ai A eh 
D ; ; n , ¥ ; tf ~~ vy 43 Harte: Bo in ty sees oh Sapna ate 
: : Fi " eS oa ee rt Be pe RY 4 es Se 
A Porn belt UY D | ; A ‘i Pi we fel T Hi 5 : Wp rh iE eee 
ee a . " 4 ‘ a “ y J ¥iey By 8 Ee a a makati wie 
7 : c eo $4 ee é & : 
¢ tie be Uy } af “ 
: ‘ 4 s A 
1 i Se a 
P ; ny : ’ 
‘ & A A 
=! wie 
q ‘ ‘ : . A y ‘ C +s 
, % 
2 co b 
Mu i ry i ‘ 
aE . if 5 1 rea i 
1 n - 4 i * 
A oe A ties , Hy i Pe A Mr 4 ae 
0 a Py “ “e : 7 C r pe oY os " 
: a ‘ o 
a : tL a i a 
A iA he t ‘ “| va * 
; 8 ry fe A "s 5 oI rt d BY x - Ty 
; $ p F Lei , esse ne 
n 1 i 
aa n % ; ee eB | 4 rs 
ans aa e J F ry . ry "yaar 
pes rs " j Fee ac) ee 
Fi ta a ' re | o ¥ ha A Leg Fin bk Msi 4 
ak: ea aie y Ayahey sane ity 
os 2 Y aa whack 8 AA ee eS 
: ¥ ? U - Wee. a ape 
- r a 1 Fy Pi $ >? er iW, i I at re Lt 
Py Cn i a ’ ra i ) LDV) ra fg het t | . 
U err A i f rl 4 : ae) Yi bk 
A eS A mi 4 AR es ce uy Se Toe ane : ve ek 
A ; as 4 ©: ? rn i Whe * nie Gees mee a 
s er N ae ‘ ieee a ated 
_ os Bers < a mi 5 if Den ie 
" rf hls aay i ; Ses Fare, ae AA 
Par , rh teats A ir , A a ct A ae A ¥ ohms Ce as i het eee jee 
Pe ee A ie 7 MF CDE a raid CY Came ac ROR y CARS eae OMA Se aes: as 
; ; 7 ? Ay me ok 3 B > RY q a, OY RO v i} at pea’ f 
= / re by i ; “iy . “hie 4 0 es ca: By Ae a pT See x $3 ine CF at a % 
- 4 F Ue | Fs F Uke, gue ge Pa 1th) § enti ; , 
P fi A 4 , a t s 
Y pet 7 > a 
b i . 
e L > 
s , F 
as oy ‘ ' 
’ M y 
‘ b 
‘ é . 
‘ J 
us oO 
2 z , ‘ 
ar) ; yi ; i 
bd 3 iD CO 
oe , F 
‘ 5 "; 
: is 
, f 
“| ‘ 
by ri He i Ps 5 ia oy sp! } y 
Fi * aE t: 9 
‘ be ae eee ne | y ie i 
’ 4 Ce ear Oi ee CL Bert PAPEL Se Rote) haat out 3 
, A ae : ar F Fa Mey ree whe 9, caer ae Pen, He £7 i helt cee te Vi 
sear ; ee ae ACE? UC 2 RR NUS Rae Ay 
4 ot ct ae é wf ‘ie . Pe a | P oP he Ve gp ignd ci | Age eS ee ‘e eee Pie 
ne ee ae , eee aie Ces IES ot Sao ted ety oan 
rs D i ; A ae y Fo oar a pie pany a D e ay eat } Y ial t as Be Rae a iss a 
A 7 ae sot g A roe ’ oe ‘ or ; Pa ar. a } A 3 ARS Fisvetoel raat 
a ‘a rey g tA Teal ee “be payy ? bite) VM or 
: ae F Ue See PS See ethan ‘SNe ae et wf ihe ate Pets are : 
: y a A Pa 4 P| r a ; 
; D P ? P F aa feva? F . ry 4 + bay ’ a : a F A) rea ‘ we ut r ‘ BURL ne CG emits ou ¥ wey 
a ff ry O i ' + , e ey 4 > het : « U : 
Pp F, F : +)’ F f . $ ? A on “4 ee”) Jaa coe anes ae a POs Ue , . ‘ ri ES 
' 7 : a z ry ¥ ‘ Ae in te re ; Ht a ny . ap ; a ot ts i Se ry oh yk De Lier on hae ty Dey he nes aes bs as fy Raton 
r “a a ; er ' oa | 5 - FP oe PF as i F aa Bry Petes aa Ui eae ee) Pre a hs a big y paps 
ra D a : iy et ee ee Lae Vee efter arr PS bef eae bet epee LY d oe nea * pasa Kegs bial 
é rw % 3 a ME ine a ha ah AA pera is a < peat ae pr Uaas bi pcr yt AN ae ihe ¢ y nfeeidies ante 
2 5 iu Po qe * : yy } or are eae p arn SON ie vhost Re i) BIT . Ee ay : a otatan 
4 As Wn} ia 3 H rot ee ees A] F Le PR any Ae "t qe. pn StL eas Y iF ae ate Ff A Bats eat ru wr petsrnne ee pod ro fas 
7 ee , ae sma, ey Se aie ie ee ge ea ty Pah Hie he Rade WSR ra aie Aa nye fades dies pcs Per ees 
oan 7 z * er Pa he Cy iy mE ee ae Ce hi a F Bi at | Hl ee ied Leet Ke pad fe ate - pcr Rie ha tr ae fai ri pars ss 
ie Agee , rar ess aga") i = Sree §, bas: Ee VAPARYE Ler a ehh Bacal ree Late cea bagi ee rete Bere ye: iat seta 
3 oan meee Le oe ; ; TEST taRay cate a ti See: me Les Si tara se oh Fs f . 
1 ae EE yy te bo hg 8 tang gy sabaress Pat wf iw She Srey a ae eR ee setae Sea PT Te Pit ol hohe Rea lok oth meen 
: : ey a ‘ a a) 8 ge AO 1 OSB Ae 3 A hee by De aes ar | 4 Pusess ” as he fh ft 
Ca rhe oa : oa Ae teen ae good) ye TMG ha RU Sd GT ae ¢ core oe alings pa On wee eT bbe vane eon rat hgh melas ges 
‘ a] net - 7 SN Wit dinias - Pra Fs pits rt Ltda Ad re Po Muiiains ‘ Bee ory am ne nee: Siete ra a 
5 r A f A F * ae F re Lae Sa) ee ep é RVs "ale iF ih ¢ si Mi eel, rh) Se eae . 
oo Fi 3 *% A ‘ - Ce a Loo 5 a b Mab i eee aR 4 ae Ete aC OGRa Roe Aare sf ara i Re Hse ar CRs Reece, a! ites poss 
A ; F ° a aot UCNNn se dary a) eae: Ty HR oth ta a the a ate rae ra Pi oe sheet peters aan re ean phe eae 
' Fr ’ rh 4 bs Us irre OMNES AL oY PR Ae a cue "ea * en aes Lea Ey ee At tated ee te AP abe 
r 5 Do Co 0 a ere, . Lr ee OR hes 1 OAL oar fet Pide- BETS Pik aeons | ee Se ve 
Cri a) , oe i s 7a nan} PSY ee 2 bs 1) er’ Pe Tees ee “or od } ie ess Pata Le ae A iy oe rae ASI Fj Pa ek Pad aon , 3 ey Wc ene Tents Sig patie 
n Fie . Pe Fi Fs a k ts Oe 0 Pe aes fi i ce ot tasih wee Porrs ar ret art yay? 4 OT bas) cS 
a D F ee 6 th 8 KR i Ge. ‘i ae ire Pie: Cert sesh bak Pod ee f nts he aby to tir hte ital Beaten 
' y y FS or OF hed ur thea eae ek SN ca: a 3 eh be bi Ree ( SAC bed (ok 
et meres 5 he bay) icc MEK ey ts briana Acne tt ed baer pet ee seta ics atte Aa cote 
. = a AE - oO re ae: ah POAC Oe Un ae erm tee vt Rots iia fy witte brbeerth bn TER fated) : Strobe Na. Senee ol 
: Fs aC ie hak iy dg fae. Pete sb Era a BSBY ARDY an Lae rh Pa ar Age ashes eek Pee wae FID 4 ra 
c toes ) LaLa Lid an ae ine ales cad Naa Ee aie, i. oo PASE ane? oo eter Lie sta 
ton ee oa os a ' p : s LG ay aus Li ye ay , Bp ke Se Weer cRe nes rs phe 4 ive ara a rae os E aad, nese else att a '” on me eats Pot 
rn i ry . ' yg U7 ‘ ; A i | PY a r) ie 7 a 
rw ¥ er ars uf en a es i een 4 oa Re AK fe es wine sie a Bs bed RG ITED Pat) Shae babel ls ep Al ee Pe ba Aah bree rN chp ee Pir ane: 
ik noes te ane Mle AEE ER gD Li NORE RRL te tei ea Rio Rr IA RY UL A SS eee ata 
a eo? Ps % Sena On a] i E Ww fe Hears Be F iy \, ay Roe at ! bt ae ee: RTA Oe é , mn be ‘ 3 on ay a i pe 
p F of Fat mere ial oO ec , oy Eee 5 eS ot , rar fee o Merc ete Pa ny te Le ae) ie i 7 Oats Sie th An aehty ch baad 
a oe Coe re ie o a ae : a ee pt aes ; th ’ Fy & - a ae PONS r Lure MS LAN re tid Non wat Sina put a eens f ee 
a : : F P ta es f 1 Bh ? have gh att a te Li 
we 4 y rise : are eet) eee ihe ‘nivdey EEE Petty, ee ray Pik ake “ie of iat fe rf ae uate pet My eel 
ae: y eee 2 can) Og AE bb Me PY Acs ik a fe Sil eh TA Ca Ps pA te oe ditty PRL aS Mace Be Shorten 
; Aan = e , Ps “ ot Rey? © ce lg eae oe ’ 17 oo ey, ‘A estatiy iy ee. Me Pay fbi cee » nis Poe Hobson 
as H , oy arte, iad ea VEY daw CS. rs A pos rt rs 
‘ 0 J *y ‘ora mo Fi a ae > a ST 0 1a = a ae} est ti or Pah re ja, steal ‘ Hite ns pees boy ee ii: rer) Mee eunGne 
: ee r i Ca 0 See) EA rag Nid hE by Ty BS PR Sete ad Che phd WR toes Spy oe ae Ps fe bs BY) Le AT 
“ me A - ae af Pe MIC f A RL Loa ak Tha area dd 4 Weibel Ae aes ase eta Poleuabee 
d : tt ey a ee et, oy at ee ras et es ce tr a A oy LT rahe Neo 
o anf i s Pr o fhe ade : ee ba a7 Sy Be ES I a of rt 4 i ‘ ae en sane Bre ee eet Aha Son 
: an het a . 4 Lb Ly Pi Als ae > 4 3 $ + 
i . as orf Be ce | aa p ae a ron OER ay gsi! eat a hae GA > het " Herel a re pacar 
7 A it P F 7] ' UME ; U it 4 ‘rn ea aa Piet haa se hae ‘; ; Rit Hh Me Gaeta ww tract dita 
: vie 0 §ty: 1 Lech AIS an h any rr bag 
ui “3 Sra F - a a EL etre ob Beta yas ee " i te ia ad ert] a Me eh RS he 
3 J t on a ee Fe At) Lees er Ly: a A bya! Fs ai a 1 ran : eae se fy ited wD y es 
t Us oF ry . a hae 1 Pes bY ae rie te Ci kd hb Se ee a Puke Pt Sn Paes A 
; p _ "ys eg” tate on ’ Se he Coe re Caner THe ad ae Dae OF ee he se Au pny aay 
et : vhs Laer, RK ray 3 ao 
P F 4 Ln ahs ‘at M a ee cer f Sy ae mn 
n Fi i) is Be Rene REG, roves ro roth 
: f + ro 7 pen vs g oe hes fontad: as) yi Ae ta Beate Rise seh ies 
r ry “ Te i) 5 ms ; a “ee fi 
eS fi Py i an | AA Sk, , noe oie Sri se ey dat 
i aria ah ho ies ae penis 
| Pa Riana ae Bona acarantett its Be 
a Ad A ra , : iC s r LT ‘ | is > Lead 
; ‘ ; OM, re ‘ a Ff f a ; FF i Cree kt ri oy AO neh ae od ” ce te 4 i y aretteey Ai) hear arr ae 
ae ae | Yan ne core Fi) eh ‘ sao re ae ; H rma ay pene ay oo ¥ eset Raa Hie 4 rady Coian Ls 
A . 2 . i a al ee! Fr Fi aula (e een - a t Phe A Yi at ge te a S ene) Vem) Sy aasiet 
o paoa ar ES i A "a ey en can ; rT ar ie Rt vi she Mi pa tees ut Ao PORE Nee pe CoP 
aR Sara na) ; | AC ae eee Tae ah Ru Ch pa ieee pe Bibs hate 3 
; ae ; F ; me He Pte ee eee aca so el x ays oa v7 wre i Toe < cy 
? a So! pte Pe ALAR Cs et Bee pe a de ee rh ‘ive 
a a | ri J U U ‘ A rs oy beh CS a aes 4 a 
¢ 4 ¢ n av é jj agri ' ar a r ie 1a. Pe) es au te) 
‘ ‘ " j 5 , Te rob ji f afi ser Ye Uta) Le LPs 277 , ¥ 
ens 8 F ; at) , : 1 ek my Pris a Pei ee ta 
‘ Ly a) : i , ; ‘ Py fey . ut r '-0e of \ 4 PP i i 5 ey pea pa Mae a fps Ag). at Cex Ca AP , i rb Pooh 
ae SES ES F Pi er Pa aol | 1 shy + c he ae b Ae Ret fare | 7 en ere Ceitt ke ae he ae 
f r] i¥ oe ‘ ; a iP % é Tat ¥ i eae peep) cc hett spre a Te fa is 
= 0 aD ae ps I eo 2 4 ye ay A) ere ae | 














NAVAL POSTGRADUATE SCHOOL 
Monterey , California 











— 


tI 


FAULT DIAGNOSIS IN DISTRIBUTED COMPUTER 
NETWORKS 


by 


Ibrahim DINCER 


we e 7? 


December 87 


Thesis Advisor Jon T. Butler 





Approved for public release; distribution is unlimited. 


1238814 





Pee LY CAS SC a ONC Petes ACs 


REPORT DOCUMENTATION PAGE 


ERO AMS ECURITN CUASSIRICA ON 1D RESTRICTIVE MARKINGS 
Unclassified 
2a SECURITY CLASSIF:‘ CATION AUTHORITY DISTRIBUTION / AVAILABILITY OF REPORT 


Approved for public release; 
Cee aru LEVON Sain) 1im1ted 


20 DECLASSIF:'CATION : DOWNGRADING SCHEDULE 


& PERFORMING ORGANIZATION REPORT NUMBER(S) MONITORING ORGANIZATION REPORT NUMBER(S) 









NPS62-87 
NAME OF PERFORMING ORGANIZATION 
Naval Postgraduate Schogl 












60 OFFICE SYMBOL | 7a 
(If applicable) 


OZ 


NAME OF MONITORING ORGANIZATION 
Naval Postgraduate School 








_ ADDRESS (City, State, and ZIP Coae) | 7b ADDRESS (City, State, and ZIP Code) 


Monterey, CA. 93943-5000 











Monterey, CA. 93943-5000 





NAME OF FUNDING SPONSORING 9 PROCUREMENT INSTRUMENT IDENTIFICATION NUMBER 


ORGANIZATION 


Bo OFFICE SYMBOL 
(if applicable) 








10 SOURCE OF FUNDING NUMBERS 
PROGRAM PROJECT TASK WORK UNIT 
ELEMENT NO NO NO ACCESSION NO 


Fault Diagnosis In Distributed Computer Networks 


ADDRESS (City, State, and ZIP Coae) 





TITLE (Iinciuae Security Classification) 










12 PERSONAL AUTHOR(S) 
jab! SCer 


13a TYPE OESREPORT 13b TIME COVERED 14 DATE OF REPORT (Year, Month, Day) $15 PAGE COUNT 
Master's Thesj Os O28 Decembe} ae es 


16. SUPPLEMENTARY NOTATION 


GOSAT=GODES 


17 —— sd 18 «SUBJECT TERMS (Continue on reverse if necessary and identify by block number) 
SUB-GROUP Preparata-Metze-Chien, Computer-Aided-Design 


19 ABSTRACT (Continue on reverse if necessary and identify by block number) 





— - 


This thesis introduces the concept of a diagnosis algorithm in the 
context of the Preparata-Metze-Chien (PMC) model. It represents a 
Computer-Aided-Design (CAD) tool for use in analyzing such algorithms. 
That 1s, with this tool, the user can establish a multiprocessor system, 
a set of test outcomes and then analyze the properties of specified 
distributed diagnosis algorithm. Examples in this thesis include a 
system in which: Ske Correct diagnosisis achieved in a small number of 
iterations. 2 Correct diagnosis ls never achieved. Ss. An 
cscillating situation exits in which, faulty processors become 


alternately enabled and disabled. 
21 ABSTRACT SECURITY CLASSIFICATION 
22a NAME OF RESPONSIBLE INDIVIDUAL 22b TELEPHONE (Include Area Code) | c2c OFFICE SYMBOL 
Pues Jon T. Butler (408) 646-3299 62Bu 


DOD FORM 1473, 84 MAR 83 APR edition may 0e used UNt.! exnausted SECURITY CLASSIFICATION OF THIS PAGE 
All other editions are oosolete 


1 






20 DISTRIBUTION / AVAILABILITY OF ABSTRACT 
C3 UNCLASSIFIED/UNLIMITED (C] SAME AS RPT 











C OTIC USERS 








@U.S Gavernment Printing Office 1986—606-24. 


Approved for public release; distmbution is unlimited 


FAULT DIAGNOSIS IN DISTRIBUTED COMPUTER NETWORKS 


by 
Ibrahim Dincer 


Captain, Turkish Army 
B.S., War Academy, Ankara, Turkey, 1978 


Submitted in partial fulfillment of the 
requirements for the degree of 


MASTER OF SCIENCE IN ELECTRICAL ENGINEERING 


from the 


NAVAL POSTGRADUATE SCHOOL 
December 1987 


TABLE OF CONTENTS 


PE OI 0) 1) 6 Oe) NI eee 10 
Pen Oe Sr WD ae ee ks he ee Re ee 10 
Ie Prepavatasletze @imem@miicrviodel... ..sugye.....-........--... 10 
Dame CCtlestcima 27... -.ennnretemennnt: > Mite: Oe... . 10 
Gre anlesatic mic Sicletenn age 2... ts <2 10 

A Oe AMES MNCS Clgn eee oc kc ede eae ake Nae a eee. 11 

See wae ce 2 ann. Sasa ea se ee i 

Gm Ie oN eens ferns cs in as. «cle I rena ean. Fe kkk il 

Fem me MOG ees Ss 4 2 Rt He 

Pile Ices Ne Sc I a oay a A ns laut edie a as. OR A ia 

Pee TOmilonmallonulestekm . 4 suai esi 46. 6G aede da oatiee se ee. 11 
PeiO mle VErINVINONIVIENT . ..665 022.0400... dR. cee cae ens 12 
yO Ieee OHI eee onic scars oo end> » « + 4 RNR ee 14 
A. PREPARATA-METZE-CHIEN(PMC) GRAPH MODEL ............... 14 
Ee ONE-S FEPMeEPAULT DIAGNOSABLE SYSTEMS ..o 020... oe, a 
IP Nc Coss na SUiniclent CONGINONS .......5.0. 08. ss ev ee eee os ly 

2. Optimal Design for one-Step T-fault diagnosability .................. 18 

Peo eOUrNTIALLY DIAGNOSABLESYSTIEMS ..................4... 19 
BONE ei Za TlON OF FAULTS <... 2:2... 0. a8. eee ee ee. 2 
Ieee ele PAPINOSADUNLY 6. 30s oie cee ss eee ces ee se ene es 21 
Pee Ate OM OSADINIEY . 222245 3S ke ee he ee es 21 

Pe cb I NOM SAR NING vas ath aed ae Saeed ck vee 6 600d a ces e ees ee eles a 
Py MMs AIR DIAIV Ghd eb ce ce ee ee ee ee ee ee ees a7 
EA et es eeu aa sa ves eee ee eae 22 
Ne et ee eis eka Ge vc we cee es 22 

28, SEB ow Gens Roel 0 ek So ee ee oe 


Tone 
Il. PROBLEM ........:..0s0000s en 24 
A. SIMPLE DIAGNOSABILITY TEST FOR MULTIPROCESSING 
SYSTEM ... 0... .0ccc este eves eset 24 
B. RECONFIGURATION ................. 0 25 
C. RELATIONSHIP BETWEEN ENABLED/DISABLED 
UNITS AND SYSTEM RELIABILITY ......)@ene. ene 26 
IV. METHOD OF APPROACH ............. ccc ccc cece cence scene ees 2a 
A. WHY A CAD-TOOL? ..............-0:000 000 v5) 
B. TOOL DEFINITIONS ........:-.6s+++-752 55) 27 
C. TOOL SPECIFICATIONS ....:.........5.,..). 28 
D. TOOL REALIZATION .......006......2-.4...0) ee 31 
V.. RESULTS 4 esses sk ees 8 33 
VI. CONCLUSIONS AND RECOMMENDATIONS ...................... 45 
A. CONCLUSIONS  ......0..4... 0.202. 00 ep ee 45 
B. RECOMMENDATIONS .........020e0es00.s245 0 ee 45 
APPENDIX A: SOURCE CODE OF CAD-TOOL ............. 0.00 cece ee eee 46 
APPENDIX B: HAND CALCULATIONS OF DIFFERENT CASES .......... 56 
LIST OF REFERENCES ».......0.s. sqeus. 1... ie 
INITIAL DISTRIBUTION LIST .....:..2:. eee ee 73 


I 
De 


LIST OF TABLES 


Ditercnimnsodels of system Diagnosis 22. nee eee aesees...... 


Menu of CAD-Tool 


¢ @© @ @ @ ®@® © @ © © @® @ © © @ @ #@ @© © © © ® @ ® @® ® ® @ @ @ @© © © @® @ @® #® © @ @© @® @ @® @® #@# @© @ @ 


Za 


4.1 
4.2 
el 
D2 
5,3 
5.4 
ac) 
5.6 
Shi 
5.5 
o 
S10 
lull 
Dele 
lS 
5.14 
aS 
5.16 
ely 
5.18 


LIST OF FIGURES 


Five Processor Multiprocessor System with 
Faulty Units and Test Outcomes .................. 


Assumed Test Outcomes in Preparata-Met-Chien 


A System and Associated Test Outcomes .............. eee 
An Example of Sequential Diagnosis Connection 
for n=14 and t=6 «eo... ee ee ee cabs eee ee 


Five Processor Multiprocessor System for Two Arrangements of 


Faulty ProceSSOIms. 444 savin. 03. cs Oatley ee 
Flow Chart of CAD-Tool.. 2.0.3.0. 0 coc. oe be 
Detailed Flow Chart of CAD-Tool ..............::...5 =e 
Cad-tool menu and test oltcomes:......5.9.......... . eee 
Case 1 initial condition ...4........3. 4. 690s4 015046504 » ee 
Case 1 first eration... ...o.ie06sas 600 oh ge... 6 eee 
Case: 1 second iteration is 2s es ks 
Case Ll third iteration <2. eiws een ena eee es + see 
Case 3 initial condition’......05. os. 6. snc ss ee 
Case 3° first 1teration: <2 een ccacee de so. os ee 
Case-.5 Second iteration <.<..04 2. Get OE oe ek ee ee 
Case 3 thitd iteration 25 ose Saas oe so 
Case 6 initial condition ...5..0.24.4.%... 3... J ee 
Case 6 first 1teratiom Sindee Sian bee a ek 
Case 6 second iteration 2..e...0.040.....0.5... eee 
Case 6 third iteration. ...... 24.4.4..0...0 4.5 5s 
Case 6 fourthiteration ...............<0:... 050) er 
Case 19 inital condition ........... 0.0... scene nee ne 
Case 19 first iteration ..........005...... 08 a 
Case 19 second iteration ............. 200. «+s sees ee 
Case 19 third tteration .......... 6. ..0..-2 0 ae en 


Sle (Case ID tresietiareinen 2. 5....8-....,.. 4? 


SJ (Case WS serereyave WS geo) 1s eee 43 
Geen Graeme MUMINGIBIFERANIOM  ¢ 2 re sc ee cc ct ec ct ttc tc ccna 43 
Pe O@ASCTOMGINMMMINGTAUOMM@s sec shee. kc cc te ck ete ee ete we eee. 44 
eMC CMCC EAU ooo eens i a ttn on oo Mee ss ow we eee a 44 


ABSTRACT 


This thesis introduces the concept of a distributed diagnosis algorithm in the context 
of the Preparata-Metze-Chien (PMC) model. It represents a Computer-Aided-Design 
(CAD) tool for use in analyzing such algorithms. That is, with this tool, the user can 
establish a multiprocessor system, a set of test outcomes and then analyze the properties 
of a specified distributed diagnosis algorithm. 

Examples in this thesis include a system in which ; 

1. Correct diagnosis 1s achieved in a small number of iterations. 

2. Correct diagnosis is never achieved. 


3. An oscillating situation exists in which faulty processors become alternately 
enabled and disabled. 
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I. INTRODUCTION 


A. NEED FOR STUDY 


The advent of inexpensive microprocessor elements has made multiprocessor 
computing networks much more practical. This fact has led to an increasing interest in 
the high reliability of such networks. The prospect of ultra reliability has inspired 
research into the use of computers where low reliability precluded its previous use. This 
includes aircraft control systems, where the Federal Aeronautic Administration (FAA) has 


specified as a standard probability of failure in a 10 hour operating period of om [Ret 


The traditional approach to computer reliability is through redundancy, where reliable 
outputs are the result of a vote on three or more less reliable outputs. In the theory of 
system diagnosis [Ref. 2}, a graph is used to model a multiprocessing system where nodes 
represent the processors and arcs represent tests between processors. One goal of the 
theory is to determine what tests achieve the highest tolerance to faults. It has been 
shown [Ref. 3] that for the same system reliability, greater throughput can be achieved 
from system diagnosis approach than modular redundancy. Conversely, for the same 


throughput, a system diagnosis approach yields greater reliability [Ref. 3]. 


Beginning with the Preparata-Metze-Chien model, many models have been developed 


for system diagnosis. The best known models are [Ref. 4]. 


I. Preparata-Metze-Chien(PMC) model: This model was used in this research and 


will be explained in Chapter II. This model is represented by Ap in Table 1.1. 


2. Perfect Tester: In this model, test outcomes correspond to perfect diagnosis of 
faulty units. In other words, if the tested unit is faulty (not good), no matter what the 
status of testing unit is (faulty or fault-free), the test outcome will be fail(1). If the tested 
unit is fault-free(good), the test outcome will be pass(0) regardless of the status of the 


testing unit. This model ts represented by Aq in Table 1.1. 


3. 1-Fail safe tester: This model never has an incorrect zero. That means that there 


might be incorrect fail test outcomes (e.g., when faulty unit is testing a fault-free unit the 
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test outcome will be 1), but there will never be any incorrect pass (O) outcome. It is 
represented by Aw in Table 1.1. 

4. 0-Fail safe tester: This model never has incorrect 1. That is, when a faulty unit 
tests another faulty unit, the test outcome will be 0. This is an incorrect pass outcome. 
However, there is no incorrect fail test outcome. The model is represented by Ay in 
maple 1.1. 

5. Ap is a model in which a faulty unit will never incorrectly diagnose another faulty 
unit. However, in this model a faulty unit testing a fault-free unit will produce O and 1 
arbitrarily. 

6. Ay is a model in which a faulty testing unit may not correctly diagnose another 
faulty unit. Test outcomes can be 0 and 1 arbitrarily. 

7. AX is a model in which a faulty testing unit always diagnoses a fault-free unit 
incorrectly, producing fail test outcome. However, a faulty testing unit produces 0 and 1 
arbitrarily for a faulty tested units. 

8. Partial tester: In this model, there is the possibility that a fault-free testing unit 
cannot correctly diagnose a faulty unit. This model is examined by Simoncini and 
Friedman [Ref. 5]. They considered the problem where system tests may be incomplete, 
l.e., that is a fault-free unit may be able to detect faulty units with percentage p (p < 
100). This model is represented by Apt in Table 1.1. 


9. Zero information tester: This model provides no reliable test outcomes. This 
model was considered by Marion L. Blount [Ref. 6]. Several different fault detection 


requirements can be addressed. 
a. A fault-free unit can fail to diagnose another fault-free unit. 
b. A fault-free unit can fail to diagnose a faulty unit. 


c. A faulty unit can give a correct diagnosis of another unit (faulty or fault-free). 


This model is represented by Ag in Table 1.1 
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Aa Aw Ag Ay Au AA Ap Apt AQ 
aij 
0->0 0 0 0 0 0 0 0 
0->0 It l l l l 1 1 
0->0 0 l X 0 0 l X 


~~ ee KR SO 
~ «M KM 


0->0 l l l 0 xX xX 


O-Fault-free unit Q-Faulty unit 


Table 1.1 Different models of system diagnosis 


All the models mentioned previously apply to a graph theoretic system. Analysis of 
such systems is typically done by hand calculation which limits the number of units. 
System fault configurations is limited to some small numbers as well. Thus, the analysis 
of such theory is difficult. Also, there is much interest in making the model more 
realistic. This, in fact, inspired the models described. For example, Ag proposed to 
model tests among processors consisting of comparing results of computations. The goal 
of this thesis is to further improve the model. Specifically, it addresses the problem of 


reconfiguration, where there has been relatively little study so far. 


B. PROBLEM ENVIRONMENT 


The fault diagnosis problem is to determine faulty processors given the set of test 
outcomes. Almost all previous studies have assumed a central diagnoser, which collects 
all of the test results and identifies faulty processors from this. This assumption 
simplifies the problem and avoids the complexities of reliable replacement. But a central 
diagnoser is also a processor, which might fail. In this case, system diagnosis may not be 
accurate. To provide accurate system diagnosis, the central diagnoser should be ultra 


reliable. This will be expensive and will require extra maintenance effort. To overcome 
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these difficulties, distributed system diagnosis is proposed. In the distributed systems 
proposed here, the hardware required to achieve reliability is simple and can be made 


ultra reliable inexpensively. 
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Il. BACKGROUND 


A. PREPARATA-METZE-CHIEN (PMC) GRAPH MODEL 


A multiprocessing system is composed of n processors. Each processor is called a 
unit (node) where a unit is a well-identifiable portion of the system which cannot be 
further decomposed for the purpose of diagnosis. Units are indicated by U1, 0 <1i<n-l. 
These units must be powerful enough to test other individual subunits. A test 
Corresponds to an arc between processors with the arrow pointing to the tested unit. Arcs 
are denoted by aj j, where i is the unit number which is doing the test, and j is the unit 
number which is tested. Each test has two outcomes, pass and fail; 0’s correspond to 
pass test outcomes and 1’s correspond to fail test outcomes. Faulty processors are 
indicated by X’s. Figure 2.1 shows a 5 processor multiprocessor system, where U2 and 
U3 are faulty. A test is meaningful only if the testing unit itself is fault-free; otherwise 


the test outcome 1s unreliable. 


au 


U 
4 


0 

U 
0 
Us 





= 


Figure 2.1 Five processors multiprocessor system with faulty units and test outcomes 
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Figure 2.2 shows how test results occur in the model we have chosen. The top arc 
goes from a fault-free node to a fault-free node and for this case a O (pass) outcome is 
always produced. The second arc goes from a fault-free node to a faulty node and for this 
case a 1 (fail) outcome is always produced. The third arc goes from faulty node to 
fault-free node and fourth arc goes from faulty node to faulty node. The outcomes of the 


last two Cases are unpredictable and can be 0 or 1 arbitrarily. 


Definition 1: The set of test outcomes aij represents the syndrome of the system; 
obviously aij can be assigned if and only if the corresponding testing link exists. [Ref. 3: 
p-848]. In Figure 2.1 the syndrome of the system for one loop will be (a01, a12, 423, 234, 
a4Q) where the left to night arrangement of the aij is intended to reflect the direction of the 
loop. Diagnosis is the process of determining the faulty units given a set of test outcomes. 


At this point, we need to define distinguishable and indistinguishable fault patterns. 


Cc 








O) 
O) 
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ww ai 


C) Fault-free 9 Fault y 


Figure 2.2 Assumed test outcomes in Preparata-Metze-Chien Model 
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Figure 2.3 A system and associated test outcomes 


Faults in units Uj and Uj are distinguishable if the syndromes associated with them are 
different. The two faults are indistinguishable if the syndromes associated with two 
different faults are the same. These definitions may be directly extended to 
distinguishable and indistinguishable sets of faults called fault patterns. Figure 2.3 
depicts a system and its test outcomes for three different cases. If Uo is faulty, the 
syndrome shown in line a is produced. If U1 1s faulty, the syndrome shown in line b is 
produced. They are distinguishable since the value ago is different. The multiple fault 
pattern (Uo, U1 are faulty) has the syndrome in line c, and since it may be the same as the 
syndrome for faults {Uo} (depending on the unpredictable values of ao1 and a12), {Uo} 
and {Uo, U1} are indistinguishable. 
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B. ONE-STEP T-FAULT DIAGNOSABLE SYSTEMS 


Definition 2: A system of n units is one-step t-fault diagnosable if all faulty units 
within the system can be uniquely identified, provided the number of faulty units present 


does not exceed t [Ref. 3]. 


1. NECESSARY AND SUFFICIENT CONDITIONS: 


In this section we investigate the relationship between n and t (the number of 
faulty units), for one-step diagnosable systems. 


Theorem 1: If a system with n units is one-step t-fault diagnosable, then n > 2t+1. 
Conversely, if n > 2t+1, it is always possible to provide a connection to form a system 


that is one-step t-fault diagnosable [Ref. 3]. 


Proof: To prove the converse, we construct a maximally connected graph, that is, 
we make a connection among all possible pairs of these n units in both directions. One 
characteristic of such a graph is that there exists a loop connecting any subset of n units. 
It is easily verified that given any loop connecting z units with all test outcomes in the 
loop exhibiting the value 0, then the z units in the loop are either all faulty or fault-free. 
In particular, if z > t+1, all units in the loop must be fault-free. Otherwise, this would 
violate the hypothesis on the maximum number of faulty units. The location of a loop of 
t+1 or more fault-free units will essentially have completed the diagnosis process, and any 
identified fault-free unit will immediately locate all faulty units through direct links. 
Since the system can have at most t faulty units, it must contain at least t+1 fault-free 


units; hence the existence of a loop of t+1 or more fault-free units is guaranteed. 


For a system with n < 2t+l units and an arbitrary connection, we show the 
existence of two distinct allowable fault patterns that may result in exactly the same 
syndrome. An allowable fault pattern for our specific case is any fault pattern with at 
most t faulty units. We can consider n as odd and even in two separate cases; but both 
cases are analogous. Assume n < 2to, with to<t. Consider the case of an even number of 
nodes. We partition the system into two parts, P1 and P2, each with the same amount of 


units t0. Suppose all units in P) are faulty and all units in P2 are fault-free. Then, all links 
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between units within P2 will have a value 0 and all links pointing from units in P2 to units 
in P will have a value 1. Since the units in P} are faulty, many possible configurations of 
values may occur. One such possible configuration is for all links between units in P} to 
have a value 0 and all links pointing from units in P] to units in P2 to have value 1. From 
symmetry, it is seen that when all units in P} are fault-free and all units in P2 are faulty, 
the same pattern of test results may occur. Hence, it is not always possible for the system 


to differentiate between the two allowable fault patterns and the system is not one-step 


t-fault diagnosable [Ref. 3: p-850]. 


2, OPTIMAL DESIGNS FOR ONE-STEP t-FAULT DIAGNOSABILITY: 


For this model it has been shown that the number of units n must be at least 2t+1 
for a System to be one-step diagnosable. Now we will try to get the lower bound on the 


number of units that concurrently test a particular unit. 


Theorem 2: In a one step t-fault diagnosable system, a unit is tested by at least t 
other units [Ref. 3: p-850]. 


Proof: On the hypothesis that the system is one-step t-fault diagnosable, we may 
assume that U1, U2,....,Uk are all the units in the system which test a certain unit Uo and 
k <t. Consider the case in which U}1, U2, ...,Uk are all faulty. The outcome of the tests 
performed by these faulty units may, of course, assume arbitrary values. Hence there is 
no reliable test being performed on Uo, and the two legitimate fault patterns (U1, U2, 
..,Uk) and (Uo, U1, U2, ...,Uk) neither of which has more than t faults are not 
distinguishable. Hence according to Definition 2, the system is not one-step t-fault 
diagnosable. Since a contradiction has been arrived at, the assertion stated in the theorem 


1s proved. 


Definition 3: A one-step t-fault diagnosable system is said to be optimal if n = 
2t+1 and each is tested by exactly t units [Ref. 3: p-850]. 

In general, many optimal designs exists for a system. To describe these families 
of designs Dy, it is convenient to designate the n units by Uo, U1, ...,Un-1, and to perform 


any computation on the subscripts modulo n. We will consider a class of designs in 
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which the testing connection at each unit is identical. In fact, whether there is a testing 
link from uj to uj depends entirely upon the value of I=j-i (modulo n). A test exists if and 
only if 1 <1<t. Preparata, Metze and Chien [Ref. 2] showed that a design Dy is an 


optimal one step t-fault diagnosable system, 


C. SEQUENTIALLY DIAGNOSABLE SYSTEMS: 


Definition 4: A system of n units is sequentially diagnosable if at least one faulty unit 
can be identified without replacement, provided the number of faulty units present does 


not exceed t [Ref. 3: p-849]. 


It is obvious that every system which is one-step t-fault diagnosable is also 
sequentially diagnosable. But a system which 1s sequentially diagnosable may not be 
one-step t-fault diagnosable. In the previous section, we have seen that nt links are 
required for a system of n units to be one-step t-fault diagnosable (design Dy). The 
investigation of sequentially diagnosable systems 1s motivated by the expectation that 
fewer test links are required in such systems. Theorem 1 1s valid for sequentially 
diagnosable systems also. Hence for any sequentially t-fault diagnosable systems n > 
ei. 


Theorem 3: There exists a class of designs with N=n+2t-2 that are sequentially t-fault 
diagnosable [Ref. 3: p-852]. 


Proof: Consider the following design. First, connect all units Uo, U1, ....,Un-1 in a 
loop such that for every 1 there 1s a link from Uj to Ui+1 (all subscripts are taken modulo 
n). Secondly, select a subset S1 of 2t-2 units from the set (Uj, U2, U3, ...,Un-2) and 
establish a link from each unit of S1 to Uo. This is shown in Figure 2.4. Let the number 
of testing signals from Sj and Un-1 to Uo having the value 0 (1) be no (n1). The following 


cases are possible: 


Case 1: ny>t. The assumption (Uo is not faulty) implies that nj > t units are faulty, 
thus violating the hypothesis on the maximum number of faulty units. Therefore n1 >t 


implies Uo 1s faulty, 
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Case 2: ni<t. The assumption (Uo is faulty) implies that, no > t-1 more units are 
faulty. If ni <t, nj+n2=2t-2 and assume nj=t-1. So no=2t-2-n1. If we put nj=r-1, then no 
=t-l. For n1 = t-2, t-3 .. and so on, no > t-1 but this also violates the hypothesis. 


Therefore nj <t implies Uo to be not faulty. 


Case 3: nj=t. Let’s consider the set S’=S1 U Un-1 U Uo for a total of 2t units. If Uo 
is not faulty, the set contains nj=t faulty units; if Uo is faulty, the system contains Uo and 
no = t-] additional faulty units, for a total of t. In both cases the set contains t faulty units. 
We conclude that all units of the system not contained with in the set S’ are not faulty and 
at least one fault-free unit can be idenufied. Therefore, nj = t implies the existence and 


identification of at least one fault-free unit. 


To locate at least one faulty unit we proceed as follows. In case 1, Uo is the faulty 
unit. In cases 2 and 3 we have located at least one fault-free unit. To locate a faulty unit 
we simply travel along the loop of testing links in the direction of arrows. We follow the 
test signals until we see a 1 for the first time, the unit being tested by this link is faulty 
[Ref. 3: p-852]. So considering all of the three cases above, we have identified at least 


one faulty unit; which is necessary and sufficient for sequential diagnosis. 


Br OO, 


ds 


OW oP 


Figure 2.4 An example of sequential diagnosis connection for n=14 and t=6 
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D. GENERALIZATION OF FAULTS 


tp-fault diagnosability: A system is tp-diagnosable if and only if the application of 
the test set identifies precisely which faults are present, provided the number of faults 
does not exceed tp [Ref. 9], (This is precisely one-step t-fault diagnosability.) 

The major part of the self-diagnosability of systems has assumed that only permanent 
(solid) faults can be present. Consideration of intermittent faults is generally difficult 
since it requires a modeling of the behavior of these faults in a system and also requires 
interactive testing strategies to detect faults. Mallela and Masson [Ref. 10] consider the 
effect of intermittent faults in diagnosable systems. The existence of both permanent and 
intermittent faults in a system, for example, affects the test outcome which is received 
after repeated applications of the test routines. This outcome may generate an incomplete 
diagnosis of faulty units, since not all the faulty units in the system may be detected. 

ti-fault diagnosability: A system is tj-fault diagnosable if in the presence of tj 
intermittent faults no fault-free unit will ever be diagnosed as faulty, and diagnosis will be 
at worst case incomplete [Ref. 4]. 

In general, the fact that a system is tp-fault diagnosable does not necessarily imply 
that it is also ti-fault diagnosable. Mallela and Mason also give necessary and sufficient 
conditions for one-step ti-fault diagnosability. 

t/s-diagnosability: A multiprocessing system is t/s-diagnosable if one can always 
identify a set of processors of size s or less which contains all permanently faulty 
processors, provided there are no more than t-faulty processors. In general, t <s, and so 
there is a relaxation of restriction in previous studies that no fault-free processors can be 


seplaced (Ref. 7]. 


E. SMITH’S ALGORITHM: 


Consider three replacement algorithms [Ref. 8] for faulty processors: 


ST1; At each step perform the tests and replace processors which fail at least one 
test, with randomly chosen spares. If all test results are pass, the system is assumed to be 


correct. 
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ST2: At each step, perform the tests and replace processors which fail the maximum 
number of tests. Replaced processors are placed back into the set of spares. If all test 


results are pass, the system is assumed to be correct. 


ST3: At each step, perform the tests and replace processors which fail the maximum 
number of tests. Put these into the SPARE-TII and replace them with randomly selected 
spares in SPARE-I. If the number of processors in SPARE-I are not sufficient, then 
choose any additional needed spares randomly selected from SPARE-IL. If all test 
results are pass, the system is assumed to be correct (initially, all spares are in SPARE-I 
and SPARE-II is empty). 


ST1 is fast but tends to replace many fault-free processors (those which fail at least 
one test by fault-free processors). ST2 replaces fewer fault-free processors, but it is 
slower. ST3 is the most sophisticated, since it tends to maintain an enrichment in the set 
of fault-free processors, and resorts to selection of suspected faulty spare processors only 


when necessary [Ref. 8]. 


d-disabling rule: Processor Uj is disabled (e.g: not allowed to participate in 


computation) if and only if Uj fails d or more tests by enabled processors [Ref. 7] 





Figure 2.5 Five processor multiprocessor system for two arrangements of faulty 
processors 


ce 


Consider the 1-disabling rule in Figure 2.5(a) and assume U2 and U3 are faulty and 
enabled. Then U4 is disabled even though it is fault-free. Uo is also fault-free and 
disabled. However, since Uj fails no test and it will become enabled permanently. It 
follows that U2 and U3 will eventually be disabled. Thus fault-free nodes U4 and Uo 
which were originally disabled will become enabled permanently. Consider the system in 
Figure 2.5(b), where there are also two faulty units, and assume the 1-disabling rule 
applies as before. If U2 and U4 are enabled, before any of the processors are enabled, the 
fail test outcomes they produce disable Uo, U1 and U3. Since all fault-free processors are 
disabled and the tests among faulty processors are pass, both faulty processors are 
enabled. Unlike the case just discussed, the system will never correct itself. Thus, a 
permanent situation exists where all faulty processors are enabled and all fault-free 
processors disabled. In the same figure, if we apply the 2-disabling rule with the same 
initial conditions (e.g: U2, U4 are faulty and enabled), the fault-free processors will 
eventually become disabled, while only one of the faulty processors will be disabled. 


Thus, the 1-and 2- disabling rule lead to an unsatisfactory diagnosis. 
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Ill. PROBLEM 


A. SIMPLE DIAGNOSABILITY TESTS FOR MULTIPROCESSING SYSTEMS 


Recall that we are interested in distributed fault diagnosis of the system, since ultra 
reliability can be achieved less expensively. The basic idea behind distributed 
self-diagnosis is that the diagnosis algorithm is executed on the remaining intact units of 
the system. In contrast to the central diagnosis which assumes an external (perfect) unit 
for computing diagnosis results, distributed diagnosis is performed throughout the system. 
First, a node is diagnosed by its immediate neighboring nodes. In a second step, these 


local diagnosis results are used to disable processors. 


To achieve distributed fault diagnosis in a system, each unit is equipped with 
disabling circuitry. Thus, testing processors can determine the status of the tested 
processor. The problem of identifying how many faulty processors can be tolerated 
before it is impossible to correctly identify them is a very difficult task in general 
multiprocessing systems. For example, in some cases as is shown in Chapter II, Figure 


2.3, the two different fault patterns produce the same test outcome (syndrome). 


The problem of locating faulty processors within a multiprocessor system by 
temporarily halting normal operation and placing it in a diagnostic mode has been 
studied using the PMC model. When the number of modules in the system is large, some 
of them will be idle at a given moment. A test may be any sort of check by one processor 
on the operation of the other, including applying test vectors and checking resulting 
outputs. In a concept introduced by Nair, Metze, Abraham [Ref. 9] called "roving 
diagnosis". One part of the system diagnoses a second part, while the remainder of the 
system continues normal operation. The part most recently diagnosed as fault-free then 
takes its turn in diagnosing other parts. Thus, there appears to be a subsystem of 
diagnosing and diagnosed units which "roves" through the system until no parts of it 
remains undiagnosed. However roving diagnosis, must ensure that first diagnosis will 
produce unique, identifiable results. The checks are performed at the system level on data 
elements that constitute the results of computations on these systems. It is assumed (Ref. 


10: 298] that each processor has a local memory on which it performs reads and writes. 
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In addition, it can communicate with other processors in the system through the buffers at 
various input and output ports. A processor cannot read or write from any other 
processor’s local memory even in the presence of a fault. A fault is any condition that 


causes a malfunction in a single processor while performing operations. 


B. RECONFIGURATION 


Definition 5: A system is c-correctable using the d-disabling rule if and only if: 
1. All faulty nodes are eventually permanently disabled. 


2. All fault-free processors are eventually permanently enabled provided there are c 


or fewer faulty nodes [Ref. 7]. 


The main goal in system configuration is to Switch-in all fault-free units and to 
switch-out all faulty units. But this switching is not between two working systems, just 
between working system and spares. The goal is not only to switch-out the faulty units 
but also keep the working system functional. That gives more flexibility to the system 
but increases the cost. The problem is to derive a distributed strategy for correct 
switching which is insensitive to the arrangement of faulty processors. Sometimes it may 
be difficult to replace a specific processor, so rearrangement of applied tests can give 
more accurate results. A flexible test arrangement will allow an approach which views 
the diagnostic task as one of arranging processors into two groups, a working group and a 
spare group. Another approach is to have three groups, one group for critical operations, 
one for noncritical operations, and one for spares. However in this thesis, we will 


consider only the first approach. 


C. RELATIONSHIP BETWEEN ENABLED/DISABLED UNITS AND SYSTEM 
RELIABILITY 


In an implementation of distributed diagnosis, to have correct diagnostics, two major 


important problems must be considered: 


1. Reliable implementation of the disabling criteria and function. 
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2. Reliable transmission of appropriate test (pass, fail) and result signals of disabling 
criteria (enabled or disabled) for system units. 
It should be noted that in distributed diagnosis, only local information is used to 
identify faulty processors. In central diagnosis all test results are used. Thus, we would 
expect distributed diagnosis to be less accurate. This manifests itself in a fewer number 


of faulty nodes which can be tolerated in distributed diagnosis. 
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IV. METHOD OF APPROACH 


A. WHY A CAD-TOOL? 


Our approach to the problem of developing diagnosis strategies is to develop a CAD 
(Computer Aided Design) tool for the simulation of different fault patterns and different 
reconfiguration strategies. Previously all studies have used hand calculations for this 
purpose. When the number of units in the system has increased to more than seven, hand 
calculations becomes complex. Thus, the user can only simulate a limited number of 
units and fault patterns. Using the CAD-tool, the user can simulate from 2 to 20 units 
with various fault patterns. The restriction of 20 units is due to limitations of the monitor 
screen. 

Thus, the tool facility gives the user an opportunity of simulating a large number of 
units and fault patterns in a system. The number of units in a network is known in 
advance and can be predefined in to the tool-program. The names and number of faulty 
nodes are determined by the user. Testing connections can be predefined by the user or 
the program. Only the test procedure (worst_case or user_defined_case) can be chosen by 
the user. Also the user defines the disabling criteria. After input by the user, the 
CAD-tool determines test results, disabled, enabled units and then displays the system in 
a control unit monitor. By using the CAD-tool, a computer network is automatically 


controlled without any hand calculation. 


B. TOOL DEFINITIONS: 
This CAD-tool is written in the C programming language [Ref. 12] using PMC graph 
model. The terms used in the program are listed below and given short explanations: 
N=The number of units in the system (may change from | to 20). 
f=The number of faulty nodes( 0 < f < N-1). 


T=The number of units which tests one unit. This number is the same for all units. 
Test results according to test connection are determined by the program reflecting the user 


desire as a worst-case or arbitrary case. 
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For the worst-case, the program itself determines all test results. That is, faulty testing 
units produce fail (1) test outcome for fault-free and pass (0) test outcome for faulty 
tested units. This information is completely opposite to the status of the units. This is the 
reason it is called worst case. For the user defined (arbitrary) case, test outcomes for 


faulty testing units (for faulty or fault-free tested units), are defined by the user. 


d=Is the disabling criteria which is defined by the user. If a tested unit has, at least d 


fail test outcomes by enabled units, the unit will be disabled. 


C. TOOL SPECIFICATION 


Figure 4.1 shows the flowchart of the main body of the system tool. As can be seen, 
the user can specify initial conditions and then allow the system to execute diagnostic 


steps one after the other. 


Figure 4.2 shows a more detailed flowchart of the program. First, the user defines the 
number of units in the system. If this number is less than O or greater than 20, the 
program produces an error message. The user defines the number and the names of faulty 
nodes. Next, the user defines T (the number of units testing one unit) and the test 
procedure (as worst case or arbitrary case). The program determines the test results and 
displays them onto the screen. The user defines the disabling criteria, the number and 
names of enabled units (all units are disabled initially). The tool displays the whole 


system in the initial conditions by calling the subroutine drawing. 
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Figure 4.1 Flow chart of CAD-tool 
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Figure 4.2 Detailed flow chart of CAD-tool 
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To see the application of the disabling rule, the user selects option #5 from the menu 
shown in Table 4.1. Then, the program determines the enabled and disabled units and 
displays the first iteration by calling the drawing subroutine. The user can go onto more 
iterations with the same conditions. After some number of iterations, the user can exit the 
program or go back to the beginning, where he/she can simulate another system with 


another conditions. 


1. INTRODUCTION 


to 


mo LOE NESE TUP 

Sob TESTRESULTS 

. SET THE DISABLING CRITERIA 
. APPLY DISABLING RULE 


NN On > W 


. EXIT 


Table 4.1 Menu of CAD-tool 


D. TOOL REALIZATION 


The CAD tool is made up of five main parts (subroutines). The first, menu option #1, 
gives a brief explanation of the program. Option #2 sets up the type of system, number 
and names of units, number and names of faulty units. Option #3 sets up T, and test 
procedure. Option #4 sets up the disabling criteria, number and the names of the enabled 
units. Then it displays the system initial conditions calling the subroutine drawing. 
Option #5 applies the disabling criteria and determines the enabled _ and disabled_ units, 
then it displays the system. In the drawing subroutine, enabled fault-free units are green, 
enabled faulty nodes are also green with X’s inside circles. Disabled fault-free nodes are 
red and disabled faulty nodes are red with X’s inside circles. Test results are represented 
by the color of testing arrows. A green arrow means a pass (0) test outcome, and a red 
arrow means fail (1). Each time, after going through each option, the menu comes onto 


the screen. So if the user makes a mistake somewhere in the program, he/she can correct 
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it easily, choosing the same option from the menu. The main part of program is very 


straightforward and just calls the subroutines according to selected menu options. 
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V. RESULTS 


Figure 5.1 shows a photograph of the CAD-tool menu. Figures 5.2 through 5.5 
shows the initial condition and three step iterations of a five unit multiprocessor system. 
In this system U2 and U3 are faulty and enabled initially and shown with color green: 
other units are disabled and shown with color red. The disabling criteria is 1 and the test 
results are the worst case. After the first iteration units Uo and U4 are disabled (red) and 
all the other units are enabled (green). After the second iteration Uj is enabled and all 
the other units are disabled. After the third iteration, all faulty units are disabled (U2, U3) 
and all fault-free units are :nabled. In this case, the 1-disabling criteria gives the desired 


results. This example is explained in Appendix B as Case 1. 
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Figure 5.1 CAD-tool menu and test outcomes 
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Figure 5.5 Third iteration 


35 


Figures 5.6 through 5.9 show another five unit multiprocessing system. In this 
example, U1 and Ug are faulty and enabled initally. Disabling criteria is 2 and test results 
are also worst case. After the first iteration all units are enabled. After the second 
iteration only U4 is disabled and all the other units are enabled. Figure 5.8 and Figure 5.9 
both are the same. This means that the system stays in that state and cannot correct itself. 


This example is explained in Appendix B as Case 3. 





Figure 5.6 Initial condition 
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Figure 5.7 First iteration 





Figure 5.8 Second iteration 
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Figure 5.9 Third iteration 


Figures 5.10 through 5.14 show a seven unit multiprocessor system. In this system, 
U1, U3, U5 are faulty units and enabled initially. Test results are also worst case and 
disabling cnteria is 2. After the first iteration, U4 and U6 are disabled, all the other units 
are enabled. After the second iteration U3, U4, U6 are disabled and the other units are 
enabled. After the third iteration only U3 is disabled. After the fourth iteration all faulty 
units are disabled and all fault-free units are enabled. This indicates the 2-disabling 


criteria works and the system corrects itself. This example 1s explained in Appendix B as 
Case 6. 
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Figure 5.10 Initial condition 





Figure 5.11 First iteration 
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Figure 5.12 Second iteration 





Figure 5.13 Third iteartion 
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Figure 5.14 Fourth iteration 


Figures 5.15 through 5.20 show a six unit system. In this system U1, U3, Us are 
faulty units and the disabling criteriais 2. [Test results are arbitrary (user defined) and are 
defined as followes: faulty testing units produce fail (1) test outcome for faulty tested 
units and produce pass (Q) outcome for fault-free tested units. In this example, faulty 
units are alternately disabled and enabled. Thus the system will never correct itself. It 


displays an oscillation of period six. This example is explained in Appendix B as Case 
19. 
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Figure 5.15 Initial condition 





Figure 5.16 Furst iteration 
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Figure 5.17 Second iteration 





Figure 5.18 Third iteration 
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Figure 5.19 Fourth iteration 





Figure 5.20 Fifth iteration 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSION 


This thesis introduces distributed diagnosis. The analysis of distributed diagnosis is 
difficult without a CAD tool. In this research, a CAD-tool has been developed based 
upon the PMC graph model. Using this tool, the user can simulate various number of 
configurations and fault patterns. The tool provides a step by step procedure for user to 
follow. In this tool, the information related to the faulty nodes (the numbers and the 
names of faulty nodes) is provided by the user. Then the user simulates the system as 


much as wanted. 


In the CAD-tool, fail test outcomes by enabled porcessors for each unit are counted 
and compared with the disabling criteria. If fail test outcomes exceed the criteria, then the 
unit is disabled. Unlike the central diagnosis algorithm which eventually settled on a final 


arrangement of processors, the algorithm denoted here develops dynamic behavior. 


B. RECOMMENDATIONS 


It is expected that this tool will be used to study optimum disabling criteria for various 
systems. For example, we hope that it will free the user of the tedium of generating 
examples, allowing him to prove properties of the system. One possibility is that it could 
be used in a knowledge base system, which would be used to prove properties of the 


disabling criteria. 
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APPENDIX A 


SOURCE CODE 


/* This menu helps the user to determine the main selections of the program: 


If the user wants to run the program for very FIRST TIME 


should choose the 


option #2.To choose INTRODUCTION is outside this restriction. */ 


char 
char’ 
int test array[206][20%% 
Lae 
PG NO en Tse, 
int response; 
menu( ) 


int response; 


primes * 
prince 
Drernen = 
Pernice. 
pierce. | 
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DG enter c. 
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printf("ENTER THE OPTION NUMBER FROM THE MENU \n\n"); 


) 


introdvction 
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eras 
*\n"); 
aos 
ae ga) F 
ore 
“uy 
ne 
ne 
ne 
cue, 
*\ oe 
eal 





eines ( "= *\n't); 
jee seta de lias EXTENSION :3299 ern): 
prant st ( * *\n"); 
pain. (| * DATE a APRIL 23,1987 ame): 


' t tt ° 
iar int f( Pte erage) ee ome eaeuitgie 1 HF HM OE HE HE IE He EH HE He ate eH We He Oe ee Ee \ Ne 


erance(”" Thismegecrameis for simulation of distributed Nea) 


printf("diagnosis algorithm in a computer network.For this 


printf(" purpose PREPARATA METZE CHIEN is used. The number 
printt(" 

printf(" of nodes in the system is restricted TO NO MORE 
meintt(" 

printf( "THAN 2@.The user enters the number of nodes,faulty 
pmantt( 

printf("nodes in the network,test procedure and disabling 
eaantt { " 

printf("criteria.The program displays the network,test 


Biarmtt( " 

printf("outcomes and shows enabled fault free nodes and 
emintt (" disabled faulty nodes. 

meaerit ft ( " N= NUMBER OF NODES IN THE SYSTEM 

pram, f ( 

maint (" D=DISABLING CRITERIA FOR FAULTY NODES 

jog gap glace 

oper t t)( ** F= NUMBER OF FAULTY NODES IN THE SYSTEM 
mormtt ( 

printf("FMAX=NUMBER OF ALLOWED FAULTY NODES IN THE SYSTEM 
Beant t (" 


printf("T= NUMBER OF UNITS WHICH ARE TESTING ONE UNIT 
} 


/* THIS SUBROUTINE DEFINES THE NAMES OF NODES AND ALSO 
NODES IN THE SYSTEM */ 
units( ) 
{ 
printf(" THE UNITS OF THE SYSTEM ARE\n\n"); 
for(i=0; i<N; ++i) 
{ 
printmewesd, "sa U' .i)s 
) 
ore Cla" yes 
printf(" ENTER THE NUMBER OF FAULTY NODES \n"); 
scanf("4d" ,&f ); 


de 


oe) 
ya); 
A) 
eine Nes 
me) 
ees) 
Nar 
age 
eld 
a3 
vile) a 
Nae) 
a) 
a): 
ly aes 
ae 
ms 
eo) 
yal ke 
Va) 
ne) 


DEFINES THE 


/* THIS TWO LOOPS KEEP THE USER IN THE ALLOWED LIMITS FOR 


FAULTY UNITS*/ 
while(f<=0 i! foN) 
{ 
Bamcr. 'N shOULD BE GREATER FHAN ZERO "); 
patio fee AND LESS THAN N \n"); 
scanf( $d", &f ); 
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PAUL, 


} 
printf("THERE ARE %d FAULTY NODES \n\n",f); 


J=1; 
/* INDICATES THE ARRAY TO DEFINE THE FAULTY NODES */ 
for(i=@; i<=N-1; ++i) 


{ 
fault_array[ij='G';/*INITIALLY ALL NODES ARE GOOD*/ 
) 
no_units_set=1; 
printf("ENTER THE FAULTY UNIT NUMBER ONE AT A TIME \n"); 


while(no units set <=f)/*REPEAT UNTIL "F" UNITS ENTERee 
{ 


scanf("4d",&i); 
while(i>(N-1) ii i<@) 
{ 
printf(" UNIT NUMBER IS NOT VALID, TRY AGAIN \\n" oe 
Scant ( sd" 21). 
} 
if (fault Janney pi! == see 
{ 
Om ci sere UNIT IS PREVIOUSLY DEFINED AS): 
Bent FAULTY, TRY AGAIN JAn\re ) 
) 


else 
{ 
fault arnaghi = —9es 
printf(" FAULTY UNIT #.4d IS  USd.\n\n" oie 
++j; 
TrnNO Un TS Set; 
} 


) 


/* THIS SUBROUTINE SETS UP THE SYSTEM 2O Bae tesla le 


Sysuset up) 
{ 
printf(" TO DETERMINE THE NETWORK ENTER ONE OF THE "); 
printf(" OPTIONS BELOW\n"); 
peinth eC Vn 
mr iMmte 1.DESIGN s\n" )- 
printed: 2.ARBITRARY SYSTEM \n\n"); 
scant ("sd ep); 
Printe, poco ae 


if (p==1) 
{ 


printf( "ENTER THE NUMBER OF NODES IN THE SYSTEM\n\n"); 
scanf("%d",&N); 
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while(N>20 {1 N<=0) 

( 

printf("THE NUMBER OF UNITS IS NOT VALID,"); 
preinvee: URY AGAIN \n"); 

scani( "4d", 2N); 

price Ned SN); 

} 
uni to), 

} 
eice 


{ 
primtsc THIS SYSTEM WILL BE DEFINED GAP eet ia ) ; 


} 


meee dio SUPROUTINE DETERMINES THE TEST RESULTS FOR THE SYSTEM.IN THE 
*WORST CASE’ ,PROGRAM DETERMINES ALL THE TEST RESULTS; FOR THE ARBITRARY CASE 
Peles esolULlo FOR THE TESTED UNITS BY ’FAULTY’ TESTING UNITS WILL BE DEFINED 


ByerHe USER. */ 


test () 


{ 
Peet T's THE NUMBER OF UNITS TESTING ONE NODE; ENTER"); 


ments "? T'\n" ); 
scanf("4d",&T); 
printf(" DO YOU WANT "WORST CASE’ TEST RESULTS?IF YES,ENTER"); 
meats ("1\n"); 
scanf("%d" ,&w); 
Pentti ( "w=2d\n",w); 
ii w==1 ) 
{ 
mon j=l3j<=l34+j ) 
{ 
for (k=0;k<=N-1;++k) 
{ 
Ll=k-j; 
bse (ie) 
{ 
L=1+N; 
} 
PC crremamraviln ==" D°) £e( fault array{l!=='B* ) ) 
{ 
test_array([k![j!=0; 


} 
ClcomemeeraulCoannaypkt==" 8") |'(fault array[1!=='B' )) 
{ 
pesteanmrayik! (| j!=1; 
} 
else 


( 
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{ 
test_array(k![j!=0; 
} 


} 
} 
else /*THIS PART user defined ARBITRARY TEST RESUE tome 
{ 
for (j= |e Gai) 
{ 
for (k=@;k<=N-1;++k) 
{ 
aie 
if ~Cl<@) 
{ 
Le ene 
} 
Pf (iaul teareay lees 
{ 
printf("TEST RESULT NODE #%d BY NODE # @da IS " koe 
scanf("4d",&test_array(k][(j]); 
while(test_array([k](j]=0 && test_array[k][j]=1) 
{ 
printf( "TEST RESULTS SHOULD BE @ OR 4 Vere 
scant (/%d",étest array[k](j]); 


} 
printi("test_arraylitd|(4d|=*d\n",k.j test array. elt). 
} 
else if ( fault_array[k]=='B’) 
{ 
test array kl (jijen, 
} 
else 
{ 
Gest array Uc =o° 
} 
} 


} 
} 
for(k=0;k<=N-1;++k) /*THIS PART PRODUCES TEST RESULT MATRIX */ 
{ 
forCjedeg«sl sets) 
{ 
printi(” |) (4¢ ", test arrayik |e 
} 
Dirane. (inn). 


) 
> ~*) ENDMCE TEST. SUBROUL INES 
/*"THIS PART OF PROGRAM IS DRAWING THE NETWORK FOR@Dis> 0A 
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#include <device.h> 
fume luge: <g1.n> 
#define resetls TRUE 


drawing() 


{ 


Br Nt, key ke, ye, ko, yo, e4,y4,%5,y995 
Mune XO yO, X 7 aval pXO> VO bales Ris 
char number[20],2Z; 
moar pi, cneta,pn1l.rho,pSsi,tau; 
sor. ang; 
m= 5.1416295; 
morn 1 ( ) ; 
viewport(400,1000,100,7080); 
eurmeorf { ); 
color( BLUE); 
clear(); 
linewidth(4); 
mmenoc(—550.0,5350.0,-350.0,3550.0); 
R= 200; 
m= © ; 
Men @COs( pi/2) ; 
y=R*sin(pi/2); 
x3=x+r*cos(5*pi/4); 
y3=ytr*sin(5*pi/4); 
x4=x+r*cos(pi/4); 
y4=ytr*sin(pi/4); 
x5=x+r*cos(7*pi/4); 
we-yrr*sint( 7*pi/4); 
x6=x+r*cos(3*pi/4); 
we=eyt+r*sin(3*pi/4); 
@or( kK=0;k<=N-1;++k) 
{ 
i=k+1; 
elm ME), 
{ 
ee leN 
) 
ang=(-3600.@/N); 
ROpeave ang,’ 2’); 
while(getbutton(MOUSE3) J]=1); 
Pe aeotonoy la=- 5 seeedisable arraylilie='D’ ) 
{ 
Golor (RED) ; 
Cireritx,y,r): 
6Olor( BLACK); 
Mevezl( xs, 75) ; 
draw2i(x4,y4); 
move2i(x5,y5); 
draw2i(x6,y6); 


Da 


} 
if(fault_array([i]=='B’ && disable array [ieee 
{ 
color(GREEN); 
CirctiC xavier: 
Color, BLACK 
MOVE2T( Xu y 54 
draw2i(x494)); 
move2i(x5,y5); 
draw21(x6,y6); 
} 
if(fault_array[i]=='G' && disable arrayia|-- oe 
{ 
color(GREEN); 
CitPcritxk, yh: 
} 
if(fault arraylil=="G'“£e disable anrayl[.i-—- 70 
{ 
color(RED); 
ClReh i yee 
} 
color( WHITE); 
cmov2i(x+30,y+30); 
sprintf(numbder,"Uted",1); 
charstr(number ); 
for(j=1;j<=T;++j) 
{ 
l=jti; 
if(l>=N) 
{ 
l=1-N; 
} 
if testearray ili cai) 
{ 
CoOlOrcRED). 
} 
if (testoarray (li j=-2) 
{ 

color(GREEN); 

} 
theta=2*pi+(pi/2)-(2*pi/N)*j; 
phi=p1/2-(pi/ Ney, 
rho=pi/2-(2*pi/N)*j; 

DSsieCoiy Ne) ; 

tau-=pi) 6; 

<4 =6* smG@ph ts) 

yi=R-r*cos(phi); 
x2=R*cos(theta)-r*cos(phi-rho); 
y2=R*sin(theta)+r*sin(phi-rho); 
K7=xK2-r*Ssin( pi /2=-psi-vauye. 
y7=y2tr*cos(pi/2-psi-tau/2); 
x8=x2-r*cos(psi-tau/2); 


De 


teen Ss, 2-psi=tau/ 2): 
y7=y2+r*cos(pi/2-psi-tau/2); 
x8=x2-r*cos(psi-tau/2); 
y8=y2+r*sin(psi-tau/2); 
move2i(xi,y1); 
draw2i(x2,y2); 
draw2tCx7-y7); 
move2i(x2,y2); 
draw2i(x8,y8); 


} 
while(getbutton(MOUSE1) J=1); 
} 
wexit( ): 


Pee this PART DETERMINES THE INITIAL CONDITIONS AND DISPLAYS 
THE SYSTEM IN INITIAL CONDITIONS */ 


disable() 
{ 
printf( "ENTER THE NUMBER OF ENABLED NODES\n"); 
scanf("%$d",&no_en_set); 
printf("ENTER THE MINIMUM NUMBER OF FAIL TEST RESULTS BY"); 
printf("ENABLED PROCESSORS WHICH DISABLE THE TESTED "); 
peaenet( “PROCESSOR \n"); 
seant ( "ed" ,&dis crit ); 
for(i=0;i<=N-1;++4i) 
{ 
disable array[i]='D’; 
} 
eounv= 0; 
Jel; 
printf("ENTER THE ENABLED UNIT NUMBER ONE AT A TIME \n"); 
while(count<no_en_set) /* repeat until all units are 
entered */ 
{ 
scanf("%d",&i); 
ee Nase <O) 


{ 
pointi, “UNIT NUMBER DS NOT VALID, TRYSAGAIN, I\n"); 
} 
else if (disable array[i]=='E') 


{ 
Beno | obi sS UNIT IS PREVIOUSLY DEFINED AS ENABLED," ); 
printf("TRY AGAIN ]\n\n"); 
} 
else 
{ 
disable array[i]='E'; 
Drinio HNABLeD UNIT #¢d@ 1S U@d\n\n",j,1): 
printf("disable array[%d]=%c\n",i,disable array[i]); 
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} 
drawing(); 
} 


/* THIS PART OF THE PROGRAM DETERMINES ENABLED AND DISABLED 


NODES AFTER THE ITERATION,DISPLAYS THE SyYSTEiia. 


apply() 
{ 
for (k=0;k<«=N-1;4++k) 
{ 
Coun?—O. 
for(j=13;j<=sT;++j) 
{ 
k=}. 
Cio) 
{ 
beaitaN: 
} 
if((test_array{k](j]==1) £@ (disable arrayivi-— ee 
( 
++count; 
} 
ite Gecunu> -atcicr ln) 
( 
fis ressarray («|= Dp: 
PRIncE Ans 
} 
) 
ifeCcount<d1s crit) 
{ 
disstes array ik|="h'5 
} 
} 


for(k=0;k<=N-1;4++k) 
( 
printf("dis res _array(i#d]=4¢\n",k,dis Pres array rap 
) 
for(k=0;k<=N-1;++k) 
( 
disable array|kl-dis resvarray pc. 
) 
drawing( ); 
princ? ! \nane 
} 
#ine lide" elon 
Finciude <stdi9.n> 
finclude <device.h> 


main( ) 
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Pe oll sel) 2 
Cunsotn ce 
Color vHl le: 
eleart i 
textport(®,3590,10,900); 
linewidth (.6,): 
while(response ]=6) 
{ 
menu(); 
scanf("%d",&response); 


if (response==1) 
incnoduccion(); 

if (response==2) 
sys_set_up(); 

if -(response==9 ) 
test(); 

if (response==4) 
disable(); 

ii onesponse==5 ) 

apply(); 


) 
printf(" PROGRAM IS OVER \n"); 
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APPENDIX B 
HAND CALCULATION OF DIFFERENT CASES 
Case 1. A five unit multiprocessor system, U2 and U3 are faulty units and shown 


underlined. Test results are worst case, disabling criteriais 1. In the matrix shown below 


testing units are placed on the x axis, tested units are placed on y axis. 


Us IB 

Uo 0 1 
Vo U4 

U] 0 0 
U} Uo 

ioe 1 
(Gays Ohi 

Ua © 20 1 
ki oe 

U4 l l 


a. first iteration with I.C 
U1, U2, U3 are enabled 


Uo, U4 are disabled 


b. second iteration 
Uj is enabled 


Uo, U2, U3, U4 are disabled 


c. third iteration 
Uo, U1, U4 are enabled 


U2, U3 are disabled 


* all faulty nodes are disabled, all fault-free nodes are enabled 
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Case 2. A five unit multiprocessor system, with U2 and U3 are faulty units and 
enabled initially. Test results are arbitrary (user defined) case and disabling criteria is 1. 


Arbitrary test results have shown underlined. 


U4 U3 
Uo 3=—sOO QO 
| Uo U4 
U1 0 0 
Ui Uo 
U2 1 l 
U2 U1 
3 «( l 
2 
U4 QO it 


a. first iteration with I.C 
Uo, U1, U2, U3 are enabled 
Ug is disabled 


b. second iteration 


Uo, U1 are enabled 
U2, U3, U4 are disabled 


c. third iteration 
Uo, U1, U4 are enabled 


U2, U3 are disabled 


* all faulty nodes are disabled, all fault-free nodes are enabled. 
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Case 3. A five unit multiprocessor system, U1 and U4 are faulty units and enabled 


initially. Test results are worst case, disabling criteria 1s 2. 


U4 Ee 
Uo 0 0 
Uo Us 
wy 1 
a6 
U2 1 0 
U2 U] 
U25) 20 1 
U32 ae U2 
U4 l 1 


a. first iteration with I.C 


all nodes are enabled 
b. second iteration 
Uo, U1, U2, U3 are enabled 
U4 is disabled 
* system Stays in that state forever 
* so system 1s not 2-fault 2-correctable 


Case 4. This system is the same as case 3. Only the test results are arbitrary case. 


U4 U3 
Uo 0 0 

Uo Us 
Ul I 1 

Ui Uo 
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U2 Jk 0 


U2 U) 
U3 0 le 

U3 U2. 
U4 l 1 


a. first iteration with I.C 


all nodes are enabled 
b. second iteration 

Uo, U2, U3 are enabled 

U1, U4 are disabled 
* all faulty nodes are disabled, all fault-free 
enabled. 


Case 5. A seven unit multiprocessor system, with U1, U3, Us are faulty and enabled 


initially. Test results are worst case and disabling criteria is 1. 


U6 es U4 
cme 1 0 

Uo Use Us 
U) l 1 0 


U)] Uo U6 
U2 l 0 0 
ED Uy Uo 


U4 1 0 1 
U4 U3 U2 
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Us U4 UB 
U6 l 0 l 
a. first iteration with I.C 


U1, U3, U5 are enabled 


Uo, U2, U4, U6 are disabled 
* system Stays in that state forever. So system is not 3-fault 1-correctable 


Case 6. This system is the same as previous case, but disabling criteia is 2. 


a. first iteration with I.C 
Uo, U1, U2, U3, Us are enabled 
U4,U6 are disabled 


b. second iteration 


Uo, U1, U2, Us are enabled 
U3, U4, U6 are disabled 


c. third iteration 
Uo, U1, U2, U4, U5, U6 are enabled 
U3 disabled 


d. fourth iteration 
Uo, U2, U4, U6 are enabled 


U1, U3, Us are disabled 
* all faulty nodes are disabled, all fault-free nodes are enabled. 
Case 7. A seven unit multiprocessor system, U1, U3, U5 are faulty units and enabled 
initially. Test results are arbitrary case, disabling criteria is 2. 
U6 Us U4 
Uo 0 ye 0) 
CoeGe Bes 
U) l ] 1 
Eine O0n macs 
U2: 0 0 
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io Uy Uo 


Lig 0 I 
Usa «Ul 
U4 1 0 Q 
U4 U3 «=6U2 
eet Q 1 
Us Us 3 
a 0: 0 al 


a. first iteration with I.C 


all nodes are enabled 
b. second iteration 
Uo, U2, U4, U6 are enabled 
U1, U3, U5 are disabled 
* all faulty nodes are disabled, all fault-free are enabled 


Case 8. A seven unit multiprocessor system, Uo, U1, U3, U4 are faulty units and U1, 
U3, U4 are enabled initially. Test results are worst case, disabling criteria is 1. 
Ue Us Us 


Uo 1 l 0 


Ug Us U5 
i 0 1 1 
Ui Ug U6 
U2 1 t 0 
U2 Ur Ug 
U3 1 0 0 
Do Uae 
Us 0 I 0 
VE wa 2 
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U6 0 l l 
a. first iteration with [LC 
Uo, U1, U3, U4 are enabled 


U2, Us, U6 are disabled 
* system Stays in that state forever. So it’s not 4-fault 1-correctable. 


Case 9. This case is the same as the previous case, except the test results are 


arbitrary case. 


Ue Us Us 
Uo ol 1 il 
Up Us Us 
Ey 0 l 1 
Ui Uo V6 
C2 Q 0 
Dec 6 
wep) it Q 


Us. ea eee 
U4 1 l Q 
Ue W U2 
Da pa QO 0 
Us a 
Ue 0 1 0 


a. first iteration with I.C 


U1 is enabled 


Uo, U2, U3, U4, U5, U6 are disabled 
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b. second iteration 


Uo, U1, U4, Us, U6 are enabled 
U2, U3 are disabled 


c. third iteration 


U4 is enabled 
Uo, U1, U2, U3, Us, U6 are disabled 


d. fourth iteration 
U1, U2, U3, U4 are enabled 


Uo, Us, U6 are disabled 


e. fifth iteration 
U1 1s enabled and Uo, U2, U3, U4, Us, U6 are disabled. 


* This is iteration #1.So system 1s not 4-fault, 1- correctable. 
Case 10. This system is the same as case 8, disabling criteria is 2 in this case. 

The test results will be the same as in case #8, 
a. first iteration with I.C 

Uo, U1, U2, U3, U4 are enabled 

Us, U6 are disabled. 
b. second iteration 

Uo, U1, U3, U4 are enabled 

U2, Us, U6 are disabled 
c. third iteration 

Uo, U1, U3, U4 are enabled 

U2, Us, U6 are disabled 
* This is I.C (initial condition) state, system stays in that loop for ever. That means 


system is not 4-fault 2-correctable. 
Case 11. This is the same as case 9, disabling criteia is 2 in this case. 


Test results will be the same as in case #9. 
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a. first iteration 
all nodes are enabled 
b. second iteration 
U2, U5, U6 are enabled 
Uo, U1, U3, U4 are disabled 


* all faulty units are disabled, all fault-free units are enabled. 


Case 12. An eight unit multiprocessor system, Uo, U3, Us, U7 are faulty units and 


Uo, U3, U5 are enabled initially. Test results are worst case, disabling criteria is 1. 
ae) Ue. 90s 
Uo 0 l 0 
Up Uz V6 


U1 l Il 0 
U1 Uo U7 
2 0 l l 


U2 Ui Us 
U3 1 1 0 

an >. ae 
0 


= 


U4 l 


Us. 204 7s 

U6 l 0 l 
U6 Us U4 

cao 0 l 


a. first iteration with I.C 
Uo, U3, U5, U7 are enabled 
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U2, U1, U4, U6 are disabled 
* system Stays in that forever. So system is not 4-fault, 1-correctable. 
When we try to simulate if the system 1s 2-correctable. 
We can easily see that Uo will never be disabled in that case. So system is not 
2-correctable either. 
Case 13. This the same as case 12, but test results are arbitrary and disabling 
criteria is 2. 
i 6 US 
iw 9 ] a 
Uo Wz U6 
ea? Oy 0 0 
U1 ld) eh 
U2 0 l 0 
U2 Ui Uo 
Zee! 1 0 
U3 “eos Ui 
U4 a ) 0 


Was Mele UD 
Ue l l 

ws, “U4 | U3 
U6 0 0 J 

Ug U5 U4 


7 l 0 l 
a. first iteration with I.C 


all nodes are enabled. 


b. second iteration 
U1, U2, U4 and U6 are enabled 
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Uo, U3, Us, U7 are disabled 
* all faulty units are disabled, all fault-free units are enabled. 


Case 14. A nine unit multiprocessor system, Uo, U1, U2, U3 are faulty units and Uo, 
U2, U3 are enabled. Test results are worst case, disabling criteria is 1. 
Ug U7 Ue Us 


Uo ] ] ] ] 
Uo Us U7 U6 


Vis <0 ] l l 
Wise or ye UE 
U2 0 0 l l 
UD eee oe Us 
er 0 0 0 l 
ba, 2. Vee 
U4 l l l l 
U4 WB WM UW 
Wis FEO l l 1 
Us U4 U3 U2 
Ue 0 0 l | 


U6 Ws-7- Ua ie 
U7 0 0 0 l 

U7 U6 Us 3 U4 
Us 0 0 0 0 


a. first iteration with I.C 
Uo, U1, U2, U3, Ug are enabled 


U4, Us, Ue, U7 are disabled 


b. second iteration 


Ug is enabled 
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all the others are disabled 


c. third iteration 
U4, Us, U6, U7, Us are enabled 


Uo, Ui, U2, U3 are disabled 


* all faulty nodes are disabled, fault-free nodes are enabled. 


Case 15. A nine unit multiprocessor system,Uo, U3, Us, Ug are faulty units and U3, 


Us, Ug are enabled. Test results are arbitrary case, disabling criteria is 1. 


Uo 


U2 


U4 


U6 


U7 


Us 


Us 
a 
U0 

i 
U1 

0 
U2 


j— 


Ua 
1 
us 
if 
U6 
0 

U7 


1 


WF 
1 
Ug 
Q 
Uo 
1 
U) 
Nt 
w2 
0 
U3 
Q 
U4 
0 
Us 
il 
U6 


1 


a. first iteration with I.C 
U1, Us, Ug are enabled 


U6 
] 
U7 

0 
Us 
Ie 
Uo 
Q 
Ul 
0 
U2 
U3 
0 
U4 


= 


Ko 


Us. 
0 
U6 
0 
uF 
0 
Us 
1 
Uo 
0 
Ui 
1 
2 


U4 
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Uo, U2, U3, U4, U6, U7 are disabled 
b. second iteration 
U1, U4, Ug are enabled 


Uo, U2, U3, Us, Us, U7 are disabled 
c. third iteration 


U1, U4, U6, U7 are enabled 
Uo, U2, U3, Us, Ug are disabled 


d. fourth iteration 
U1, U2, U4, U6, U7 are enabled 


Uo, U3, Us, Us are disabled 
* All faulty nodes are disabled, all fault-free nodes are enabled. 
Case 16. This the same as previous case but disabling criteria is 2. 
a. first 1teration 
Uo, U1, U2, U3, Ua, Us, U6, Us are enabled 
U7 is disabled 


b. second iteration 


U1, U4, U6 are enabled 
Uo, U2, U3, Us, U7, U8 are disabled 


c. third iteration 


Uo, U1, U2, U3, U4, U6, U7 are enabled 
Us, Ug are disabled 


d. fourth iteration 
U1, U2, U4, U6, U7 are enabled 


Uo, U3, Us, Ug are disabled. 
* All faulty nodes are disabled, all fault-free nodes are enabled. 
Case 17. This case is the same as case 15, but disabling criteria is 3. 


a. first iteration 
all nodes will be enabled 
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b. second iteration 


U1, U2, U4, U6, U7 are enabled 
UO, U3, Us, Us are disabled 
* All faulty nodes are disabled, all fault-free nodes are enabled. 


Case 18. A nine unit multiprocessor system, U1, U3, Us, Ug are faulty units and U1, 


U3, Us are enabled. Test results are worst case, disabling criteria is 2. 


Us U7 U6 aes 


Uo ] 0 0 I 
Uo Ug (U7 U6 

Ul l 0 l l 
U; Uo Us U7 

U2 l 0 l 0 


U2 Wry Vo Ue 
U3 I 0 I 

Pane UU; #£4zUO 
U4 l 0 l 0 


SS 


Us U3 U2 Uj, 


Us U4 sua U2 
U6 ] 0 l 


Ue Us Ud WG 


— 


U7 0 l 0 l 
We OF amen Ok} U4 
Ug ] l 0 ] 
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a. first iteration with I.C 
Uo, U1, U2, U3, Us, Ug are enabled 
U4, U6, U7 are disabled. 
b. second iteration 
U1, Us, Ug are enabled 
Uo, U2, U3, U4, U6, U7 are disabled. 
c. third iteration 
U1, U3, U4, U5, U6, U7, Us are enabled 
Uo, U2 are disabled, 
d. fourth iteration 
U3, Us are enabled. 
Uo, U1, U2, U4, U6, U7, Ug are disabled. 
e. fifth iteration 
Uo, U1, U2, U3, U4, Us, Ug are enabled 
U6, U7 are disabled. 
f. sixth iteration 
U1, Ug are enabled 
Uo, U2, U3, U4, Us, Us, U7 are disabled. 
* System is not 4-fault 2-correctable. 


Case 19. A six unit multiprocessor system, Ui, U3, U5 are faulty units and only Uj 


is disabled, all the other units are enabled. Test results are arbitrary case, disabling 


eritenials 2: 
Us U4 
Uo Q 0 
Uo Us 
a) 1 
Ui Uo 
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oo 


Ue Q 0 


eli 
3  ! i 

U3 (U2 
we 0 

U4 U3 
5 al | 


a. first iteration 
Uo, U2, U3, U4 are enabled 


U1, Us are disabled. 


b. second iteration 
Uo, U1, U2, U3, U4 are enabled. 


Us is disabled. 


c. third iteration 
Uo, U1, U2, U4 are enabled. 


U3, Us are disabled. 


d. fourth iteration 
Uo, U1, U2, U4, US are enabled. 


U3 1s disabled. 


e. fifth iteration 
Uo, U2, U4, Us are enabled. 


U1, U3 are disabled. 


f. sixth iteration 
Uo, U2, U3, U4, U5 are enabled. 


U] is disabled. 


*That is I.C state and system oscillates and returns to I.C state in every six iteration. 


v1 


10. 


ike 


LIST OF REFERENCES 


J.H. Wesley, et. al., "SIFT: Design and analysis of fault tolerant computer for 
Aircraft Control," Proc. of IEEE, Vol. 66, No. 10, pp. 1240-1255, October 1978. 

F.P. Preparata, G.Metze, and R.T.Chien.,"On the connection assignment problem of 
diagnosable systems, IEEE Trans. on Comp., Vol. C-16, pp. 848-854, Dec. 1967. 
K.Y. Chwa and S.L.Hakimi, "Schemes for fault tolerant computing: A comparison of 
modularly redundant and t-diagnosable systems," Inform. and Control, Vol. 49, No. 3, 
pp. 212-238, June 1981. 

Arthur D.Friedman and Luca Simoncini, "System level fault diagnosis," IEEE Trans. 
on Comp., Vol. 13, p. 47-2, March 1980. 

Simoncini Karunanithi and A.D. Fnedman,"System diagnosis with t/s diagnosability,” 
Proc. of the 7 th Fault-tolerant Comp. Symp., pp. 65-71, June 1977 

M.L. Blount, "Probabilistic Treatment of Diagnosis in Digital systems, Proc. 7th Intl. 
Conf. on Fault Tolerant Computing, pp. 72-77, June 1977. 

J.T. Butler, “On the design of distnbuted diagnosable multiprocessing systems,” 
Naval Postgraduate School Monterey, CA, research proposal. 

A.L. Hopkins, T.B. Smith, and J.H. Lala, " FTMP-A highly reliable fault-tolerant 
multiprocessor for aircraft," Proc. of IEEE, Vol. 66, No. 10, pp. 1221-1239, 

October 1978. 

R. Nair, G. Metze and J. Abraham, "Design Considerations for Fault -Tolerant 
Distributed Digital Systems,” unpublished manuscript. 

S. Mallela and G. Masson, "Diagnosable systems for intermittent faults,’ IEEE Trans. 
on Comp., Vol. C-27, pp. 560-566, 1978. 

Stephan G. Kochan "Programming in C," Hayden Book Company, 1983. 


72 


i 


10. 


INITIAL DISTRIBUTION LIST 


No.copies 
Defense Technical Information Center 2 
Cameron Station 

Alexandra, VA 22304-6145 


Library, Code 0142 Z 
Naval Postgraduate School 
Monterey, CA 93943-5002 


Department Chairman, Code 62 l 
Department of Electrical and Computer Engineering. 

Naval Postgraduate School 

Monterey, CA 93943-5000 


Dr. Jon T. Butler, Code 62 BU 5 
Department of Elecrical and ComputerEngineering. 

Naval Postgraduate School 

Monterey, CA 93943-5000 


Dr. Bruno O. Shubert, Code 55 SY l 
Department of Operational Analysis 

Naval Postgraduate School 

Monterey, CA 93943 


Dr. Dana E. Madison, Code 52 l 
Department of Computer Science 

Naval Postgraduate School 

Monterey, CA 93943 

Dr. Joo Kang Lee l 
POSTECH Research Institute of Science 

and Technology 

PO.Box.125, Pohang City 

Kyungbuk 680 KOREA. 


Director of Research Administration, Code 012 1 
Naval Postgraduate School 
Monterey, CA 93943-5000 


Kara Kuvwvetleri Komutanligi l 
Egitim Dairesi Baskanligi 
Bakanliklar, Ankara, Turkey . 


Kara Harp Okulu l 
Bakanliklar, Ankara, Turkey 


73 


ee 


13. 


fo 
= 


Lo 


> 
er) 
e 


Muhabere Okul Komutanligi 
Mamak, Ankara, Turkey 


Capt. Ibrahim Dincer 
Muhaber okulu 

Ogreum Kurulu 
Mamak, Ankara, Turkey 


Lyg. Mustafa Paktuna 
Marmara cad. No:158/6 
Kocamustafapasa, Istanbul, Turkey 


Dr. Andre von Tibborg 
Code 1133 

ONR 

800 N. Quincy 
Arlington, VA 22217 


Dr. George Abraham 
Code 7500 
NRL 

4555 Overlook Ave. S.W. 
Washimetons, De 206 a5 


Dm EOUrec iid 

Oh = 207 

800 N. Quincy Ave. Room 811 
Amiingtion.e Vaec2ciy 


74 




















mp eiied De hot s s 
fase haedg ats eae wnat hui 


ae Ca ree 
depleted EI +4 


eh wee} ns 

Bit OF. 4M bs en 

ON a Br Se aray 

erro Sree 
he : 

" elie y ge 


Serres op 
er we Searcy rt 
> a or Set ae 

hi Fed be ae Re yeh 


Fault wel in distributed computer 


| {| ft! | 


ex ry 2 me: me 





eb oe 


ere tos Sera 
PE RS ee ee | 
aot 2 rr ka 


rear o Ne 
te Te oe 


& gy Pay 
peer ny Han AEX P 


Borirtety 


Sec aisretse. 
RNs Orin? ae re 


Seis domerg 


elepawere > OM oh 
Pherh peak 


Perret ranged f et 


MRAM ATTA 


#3 2768 000 78140 a 
DUDLEY KNOX LIBRARY 


pore roy sa 
oe ier re . 
pacar mere hat oe | nt nN ney 


nig nA Ser a ry ae 


3 wh ten 34 
Ped ee rr) A ved Are ey 9 
i be abes at de 
Phe ob ee a bape hoop. Nad fi. 
Sato oe iy were eH arass 


ant  Peedoveatel erg A) 
Mey Deena Goby! arr Ae’ 3 are 
Pd ORAM MR 4 Oo “0 
Ar pre Meter carr: oot 
eed uchipedeveseonayy he of 
Oe ed, ee th Se et ent 
ON rr ee ts 
Leetealh LAF hipaaga oc ebepeeenrtag 
MiveteP lenith 





nae TN , 
be Dates ye 


epee or wn 





iN iro LY Ve 
Pos try mpeg 


eRe ae i 7s 


Seal Sede 


sO : thn 
Eee en s earth ’ 
ee 


amar che. pit oe a rave 
PPA ad a " ere ae re, oat 





Prt As + Ory 


rae +t bbe (<7 ree ey 
yeh ey oct 


watt ie fo Ae 


Ke pttnas fy 10-day A) bs eres 
ee eae Ly), he des 


Ss et freer Aihe ry 


pre oer ore i revs LE Foal oe © tre 
pan hed reo ri 





yr arr} task 
ry ry Se YC 
et ee eet yh os 
LB are eke 


an Reet ire 
poet cee Srey 
Adcad ds Wud ulusis ahs tatiana had pet ee Avent dubia! 
Pelt hike te sale Oe ceo l oe eee 

of dete Batre HF phihl B2¢ errs A rr at 
baba in deck elatabbek. alti 
ORIEN Dy 
tM Bias fereiny oy hare ea HY 

m9 nae Sor rr ak bs 


iy hee - 


" mH yer aes Cz 
Deg a le 


FA yD orients 9 hae s Fe te aCe 


LP te ce 





M6 topeaer, hah Bom Pers 4 os ‘ue ow 
i OT pe ey 
Cay eT #: aye! 


Af LL oie) 
pate STR oe Se 


po on + tive i 
Le aa PAS eaten ae Satee 

eo plete biegty PSs aaa * meu 
7) a3 OH eed 


pre dnalpatone set wr Pe ee Te 
Ce Roner ML eed 
Dey y oeqaormn TT pry tet UT ers 

Pte {tres Prey mre 
a ee Te ae 
bn 4 sor on et recrae Ye 
pit brie anal 

S Aarts fhe rer 
ere woe rae * 
< Pre rer ser yi 
ery Serer re al dale oe Yoke ee eps 
ore bith ch oth Oe ateke Val gale, § GH 
ey yep co erp 
a = dra@ae ease nenee atari 
Bway Phys ry) pros fata pe rs 

Oe a Lele 
DY eet Bote Setied 
ep oyrupee aw ea ee ev Fe et Ded it Ce ve 


mate Dre 
tyes 
is Fak if 


£4 pag) BE OSA.:¢ 6 
in TL Sal eg 


ee LY 
te "Cine. Ue 


tee a A 
Ty arr ie eS 


rey yr: Sree S “ 


ator poeliay 
ea roe 
A 





ey aA ferent ry ee ae et a! 
Pre ey ee 
ee Oe eg ee 

Mesieaie ot oe Oy a 


Ls False “ee 7 


panied ri re cel Ate tyler pt 
4 fon pre aries 





RESUME Pare iristets 

wee ree | a4 Lee oi tare ots a? re Pls rarer 

erie Te mt ae el Oe STS Y cer ee eae 
4 


FoR ; Gare patie ate, 
ines eck Sieh hbo 


Are wet t 


aes Pe erent 


ok ee rv 
eet a o ot 


eae ae 


fear A af of PoP oy PA Mb f- 
Soot Se 


beri mds re riya wh Bon re 
ey 1 ee) 





aver gorge 


owed [SPR Oy 7 ee Som eI re 


feet ny eran 


als neers 
gato iA 


Meme iia 


Cyr a ee ee 
ee shat 7 re 


ere argent " 


Sie ee ee 








r 
heaps nears how het eb VP a rs 4 
eres oat fryy Po pavews§ Lenk 


“0 led Sarre i bacin| 7 


re pees 


Sead ao hus te Pr ree bn rove Mpa, cD 
ree eee ere Re Sik owed v Adaya lhe) 
Se ecete stad P27 


Raver iy. artes 
Perr | 7 rh © ehtoue! 


ae reno ie Peet 
7 ott ols Witte a Chee ae 
Ora a wr A 
Ary Seca elidel oe ove 
Pe Eder ce) ten ded 
errs OOS Siena 
Sao Bye ABS FA ne peat he 
PRS Peon oY +x 
Tye pate te fate 
Tr te Po): 


CARD as! ater 


iD ae id Set barks a rt) 


ck ntl ere 
Cae ra 





7 ped Argo Caron I 





ep A of a r : 
Coal oo Sy ery tee ies 
: S TT ed etal ald 
a ry Sie paren ; 
, art) dend p hil Lanka 
oer nae a Prt Yh 


; Sarit ws & 
tee he Peheah et eS 








Nl mA +7 
rrr) ote 
+e oases ie a 

gh 1 att te 7 ar ear ee 
artis pra 
Pre) Cina oy 


eet eet 
SpaA an 


od as Se ath 


ars Peet] a dics. are 
Ree. a pe Pats 
Eyes iy oye. Bee a ae rn. 
2 edbeter errr oe by: 


Pie Merde a i 


Irae mata 
eLearn 





areel € Fa 9 oo 
Pier eee 
CM Ee * Del rate 
toe ne 4 uae: 


ie OL Cte 
eRe erty 

Pare EAM 
Pela and 


eee ~ Sy ie 


eee Sa 
rahe Saat Ad 


wrier 











7 pelt ae Srerk aia 


eee nea Ms 


ce st 












Att * Rigo Ri she 
rac he Ea 


v7 | 
aa sie i 


od ial A 


e 


tr hie rad p 


ig orient 





Pe ety Ae a 
eS ES Co Siren! 
A ttt Rg. by ry Ieee 


aT oR BA ee 
teeta 


and cr 
eae me 


Tp fe ye es 
brig Sorat See 


, Pare la i aa | 
re ‘ es 











“pe Rv ro Ys nl OS din: of 
eS ry pe yar er or 


 stinhtt 


Feeds 
Rare cow) 


EER 


ee ens “ ert 








Nati f ah tle» a P 
















Renee Faure rh i 













= 


a 





ey 
Sua? 


Ge 


a 


os 


o 

oy 
s 
hu 


mh oe 
os 


> ligt alla fe te 
a. 
a 


; 


~% 
a 

(A Yes 
ad 







ie 


ae 


Pitot“ 





4 
is 


4 





» P* a 
TJ 
a ae 
ree) 


ry 


ct 


hn es 
5 








a 
y 








« 


5 
‘\ 
eo 
rad 
ae 
Ys 


» 


va 
Pacates ; 


vs 
a 
ft, 


a 


= 


Fie 





rs 





ag 
on 
Oe 


es 


s 


- 
ars 

i taka 

aed 


oo 


Wy 
e 
Py 





“eo 


ae 


Ey 






1, 


Peo 
a 
en 

7- 
Be Pr ge 
gr, 






i er* 
* Nabe 


ote 





th | 








‘ 





a 
yi 
s 


x 


= 











es 
2 









ISI 
. 







-” 
P 


ri teat ats hsb 


4 






y 


sie 





” 
Cet 


” 


Cw a 


ds 
4 
eS 


uy 






Eo 


een, 


5 





an * > et 


PSs 
rf 
Ys 


* 


oats el 
et ae Ei 





“ae 
Bra 
CG 





ss 
4 
a 
*g 
eS 





~ 
an 
ad 


ed 
er 





3 


2 






ete 


“« 





oY Oe 
To 


Thich ue 


oe Sake cp 





tat 













5 


rr 
Ser eh 

BY hs Be 
6 Sie ~~ 


‘i Seek bf ai 


) 
Pa 


yO 44 ar fers rs 
Beh a 


aA 7 






ha Seen 


py Bt ee eae 








ers 


M oe a Wak ee 


ath vores 


a Ha 


0 


ma pas ioe 


een a ot 


chivae ty g Ba 














By ta Ay 
Abd yeh Bat bays, 


a *e RR ee tray 


Ped n pe eh 
igige Pat ate 


An Sv ytd va 
ae ee. ea 


Se rh ei 





maps? Le 
wits es nity 


Lit Ny a Otel Tene 


sais is 


pwd oe ao 










Hee aye Stout 
ny ae 
ane 


titel 
ene 
pea shi 


ys san 


bite t eR’ 


rt 


Se Soe 
















Se head. i uy 
ae Dy re * Sit 


- 4 re . pana 


ae 
ae My aa 
an +. Fi AF TE 





mee Aah) a 





Adah Fi 
an & + a ee 
re ety 0 Rt sats he 


og Pet aha 
aed pe we bier a) e des re “FS 


fe oa Be 





vy Gs Pht) | 


Ad ee | ote 
i ee] ie bee ae in | A ae | ts wea | 

boli 8 ceyernleh j 
tha en em 
mck es renee: 


rest a 

ahs Eeacats a iah 

w”AP aa my 

Sek ty ie 
ret Lode) 

Sts ra, 

say Sl 








rae ou GC 


ae 


eet ey ™ oy a, ae 
mil hat Pg wath oe tha oe) 
5 PB ap ighel Roe ath eran 

‘ ie 


eye 6 
hs +o) es! het 


ah? aa yah Ho Nae 


eer tes Paha tees 
. ya aes ( 


a ah ital Ui mS ¥ 


Kd iP 
coe 


Y pT Ry a 0 ks a vy ‘i 
SA th nie ns ayy 


his aed Tete 


arb ees 


J Pa? 
Por ay ae shy A 
re Deoaet e 


vat) A a h alah 
eRe mi eres Ahi 





as aise om 





Tulare | Ih 7B 

uel ok RT 3 
APG YAEL ONY Veg vw. 
any baie . A 


Ce) am v ea ey . 
ay nae biaeds th he 


ah he we are 


at Bf cae ka re ‘Ty 
Wehr wr YseTY MS 


SA ead 
J Le MoU ee 
‘gel as y 4p RA ta 

ve Uh me ee ae Bd 


rr eh th biel 


ars ite She a Ot Path P 
ae pa Seen oe pa pains ea 





teal base SS “ys a 


7 
M es 


ce Bea 


ath 
ey Vv wy bi tae 
Si casts) il ai 
TRO ry 


ra te Rak 


amen 
1" % baie 


ae phe 
peti kd A 


a eas 5 hk Me! rth LM ae 





j 
oe + ra € 
ae es et 
~ ‘ xO hy i me 


ad . 
Sree eas Lat YP 
ee ee | % °F 
ah wate te RN LIN er 


ah Pea a 


SMH r an 





ho le toate 
bf a a ke A Se) 





hae an Se 


tf + 
te a +i i ae _ 








on wee ww 
$6 Pog hae A 


has Lah ML da ret y 
x! Re Wa cea od 

See ea kD 

9 er) Drewiner 

testa lati eew 





he is y WENT 
oe Oe ewe & 
A eee ae eS | ea tes 
OL ole a | w Scare 
a ee 2 ws 
ae iw) 
aD iH he 
aed thm eet ati te k 
pate uy FOG eee v* meh She We ea A 
oe SRST TEENS its ede 
eet Mieke Me A Led 
Ete Saath Ale | 
we eit Ls ye Lia 
oi 
Ad dah OMe J Dik oh 


raters 


eh oe Boh | 
Jeera , 


iD fe Sa cate 





TaN re 











Cw) 
A ee a at vi 





Ad Wet St as 








eae 


a 


Pa 


